Latent Space Oddity: On the Curvature of Deep Generative Models

This repository provides a clean, well-documented, and mathematically faithful implementation of the paper "Latent Space Oddity: on the Curvature of Deep Generative Models" (Arvanitidis et al., ICLR 2018).

The work explores the Riemannian geometry induced by stochastic generators (VAEs) and proposes an expected metric that captures data distortion and uncertainty, enabling the computation of meaningful shortest paths (geodesics) in the latent space.

Variance Extrapolation Comparison	Latent Metric Measure Visualization

---

🔬 Mathematical Core

The Expected Metric

A stochastic generator $f(\mathbf{z}) = \boldsymbol{\mu}(\mathbf{z}) + \boldsymbol{\sigma}(\mathbf{z}) \odot \boldsymbol{\epsilon}$ induces a stochastic Riemannian metric. The paper identifies that the expected metric $\overline{\mathbf{M}}_{\mathbf{z}}$ provides a deterministic approximation that becomes exact in high dimensions:

$$\overline{\mathbf{M}}_{\mathbf{z}} = \mathbb{E}_{\boldsymbol{\epsilon}}[\mathbf{J}_{\mathbf{z}}^\top \mathbf{J}_{\mathbf{z}}] = \left( \mathbf{J}^{(\boldsymbol{\mu})}_{\mathbf{z}} \right)^\top \left( \mathbf{J}^{(\boldsymbol{\mu})}_{\mathbf{z}} \right) + \left( \mathbf{J}^{(\boldsymbol{\sigma})}_{\mathbf{z}} \right)^\top \left( \mathbf{J}^{(\boldsymbol{\sigma})}_{\mathbf{z}} \right)$$

Ensuring Proper Geometry

Standard VAE variance functions extrapolate poorly in regions without data. This repository implements the RBF-based precision model:

$$\boldsymbol{\beta}_{\psi}(\mathbf{z}) = \frac{1}{\boldsymbol{\sigma}_{\psi}^2(\mathbf{z})} = \mathbf{W}\mathbf{v}(\mathbf{z}) + \boldsymbol{\zeta}$$

where $\mathbf{v}(\mathbf{z})$ are radial basis functions. This ensures that the variance increases (and thus the metric "walls" up) as we move away from the data support, forcing geodesics to stay within meaningful regions.

---

📂 Repository structure & Paper Mapping

Paper Concept	Code Implementation	Description
Stochastic Generator	generative_models.VAE_model	Implements the mean and variance decoders.
Expected Metric	manifolds.MlpMeanInvRbfVar.metric_tensor	Computes $\overline{\mathbf{M}}_{\mathbf{z}}$ using Jacobians of $\boldsymbol{\mu}$ and $\boldsymbol{\sigma}$.
RBF Variance	reproduce_paper.py (Training)	Fits RBF network to empirical VAE variance.
Geodesic ODE	geodesics.geodesic_system	Implements the 2nd order ODE for shortest paths.
Solvers	geodesics.solver_bvp, solver_graph	Boundary Value Problem and Graph-based initialization.

---

🚀 Getting Started

Prerequisites

• Python 3.8+
• PyTorch
• Scipy, NumPy, Matplotlib, Scikit-Learn

Installation

pip install torch numpy scipy matplotlib scikit-learn

Reproducing Results

To train a VAE on synthetic data, fit the RBF variance model, and visualize latent space geodesics:

python reproduce_paper.py

This script generates:

• ELBO Convergence Plot: Training progress of the VAE.
• Metric Measure Visualization: Heatmap of $\sqrt{\det(\mathbf{M}_{\mathbf{z}})}$.
• Geodesic Plots: Shortest paths connecting points in the latent space.

---

📝 Citation & Lineage

Original reference implementation: georgiosarvanitidis/geometric_ml

@inproceedings{arvanitidis2018latent,
  title={Latent Space Oddity: on the Curvature of Deep Generative Models},
  author={Arvanitidis, Georgios and Hansen, Lars Kai and Hauberg, S{\o}ren},
  booktitle={International Conference on Learning Representations (ICLR)},
  year={2018}
}

---

Maintained by prikarsartam.