Learning Flat Latent Manifolds with VAEs

Nutan Chen, Alexej Klushyn, Francesco Ferroni, Justin Bayer, Patrick van der Smagt

TL;DR

Standard Variational Auto-encoders (VAEs) often struggle because the "straight-line" distance in their latent space doesn't actually correspond to how much the data changes in the real world. This paper introduces the Flat Manifold VAE (FMVAE), which uses a novel regularization technique to "flatten" the latent space, ensuring that simple Euclidean distances accurately reflect true data similarity.

What Problem Does This Paper Address?

In machine learning, we often want to measure how similar two data points are. VAEs help by compressing high-dimensional data (like images) into a low-dimensional "latent space." Ideally, if two points are close in this latent space, they should be similar in the real world.

However, the math behind VAEs (specifically the decoder) often creates a "curved" geometry. A small step in one part of the latent space might result in a massive change in the output, while the same step elsewhere might do almost nothing. To fix this, researchers usually have to calculate complex "geodesics" (the shortest path on a curved surface), which is computationally expensive. The authors wanted to find a way to make the latent space naturally "flat" so that we can just use a ruler (Euclidean distance) and get the right answer quickly.

Methods

The authors propose the Flat Manifold VAE (FMVAE). Their approach relies on three key pillars:

  1. Riemannian Metric Regularization: They define the latent space as a Riemannian manifold. The "stretchiness" of this space is captured by a metric tensor ($G$). To make the space flat, they add a penalty to the loss function that forces $G$ to be a scaled identity matrix ($G \propto I$). This essentially tells the model: "Every direction and every location in the latent space should represent the same amount of change in the output."
  2. Latent Mixup: Regularization usually only happens where there is data. To ensure the space is flat even in the "empty" gaps between data points, the authors use mixup. They interpolate between encoded data points in the latent space and enforce the flatness constraint on these synthetic "in-between" points.
  3. Hierarchical Priors (VHP): Standard VAEs use a simple Normal distribution prior, which can be too restrictive. The authors use a more flexible Variational Hierarchical Prior, which gives the model the "breathing room" it needs to rearrange the latent space into a flat configuration without losing reconstruction quality.

Datasets

The model was evaluated on four distinct types of data:

  • Artificial Pendulum: 15,000 16x16 images of a simulated pendulum used to visualize the geometry of the learned space.
  • CMU Human Motion Capture: 50-dimensional vectors representing joint angles for movements like walking, jogging, and kicking.
  • MNIST: The classic dataset of 70,000 handwritten digits.
  • MOT16: A challenging real-world video tracking benchmark used to test how well the model handles object re-identification in complex scenes.

Key Insights and Ideas

  • The "Aha" Moment on Flatness: A "flat" manifold is isometric to Euclidean space. By forcing the decoder's Jacobian to be constant, the authors effectively turn the latent space into a coordinate system where "distance" actually means something consistent.
  • Unsupervised Tracking: One of the most surprising insights is that this unsupervised method can compete with supervised ones. In the MOT16 task, the FMVAE learned to identify people across frames just by looking at latent distances, nearing the performance of state-of-the-art models trained with explicit labels.
  • Efficiency is Key: Because the space is flat, you don't need to solve differential equations to find the shortest path between points. You just subtract one vector from another. This makes the model practical for real-time applications like robotics or live video analysis.

Key Results

  • Metric Consistency: On the pendulum dataset, the FMVAE produced "equidistance" plots that were nearly perfect circles with a uniform magnification factor, whereas standard VAEs produced distorted contours and highly variable magnification.
Pendulum dataset latent space comparison
  • Smoothness: In human motion interpolation, FMVAE significantly outperformed standard VAEs. The Euclidean straight-line interpolation in FMVAE closely matches the true geodesic path, producing much smoother transitions between poses.
Human motion dataset latent space comparison
  • Tracking Performance: When integrated into a tracking pipeline (SORT), the FMVAE reduced "ID switches" (losing track of a person and assigning a new ID) by over 2x compared to a standard SORT baseline. It achieved an IDF1 score of 64.2, which is remarkably close to the supervised DeepSORT (64.7).

Limitations and Future Work

  • Hyperparameter Sensitivity: The model introduces a scaling factor ($c$) and a regularization weight ($\eta$). Finding the right balance between "flatness" and "reconstruction accuracy" can require tuning.
  • Global vs. Local Flatness: While the model strives for a global scaled identity matrix, complex data topologies might still force some curvature in very high-dimensional settings.
  • Future Directions: The authors suggest that this "flatness" could be useful for reinforcement learning, where agents need to understand the distance between different states of the world to plan effectively.

Summary updated

Abstract

Measuring the similarity between data points often requires domain knowledge, which can in parts be compensated by relying on unsupervised methods such as latent-variable models, where similarity/distance is estimated in a more compact latent space. Prevalent is the use of the Euclidean metric, which has the drawback of ignoring information about similarity of data stored in the decoder, as captured by the framework of Riemannian geometry. We propose an extension to the framework of variational auto-encoders allows learning flat latent manifolds, where the Euclidean metric is a proxy for the similarity between data points. This is achieved by defining the latent space as a Riemannian manifold and by regularising the metric tensor to be a scaled identity matrix. Additionally, we replace the compact prior typically used in variational auto-encoders with a recently presented, more expressive hierarchical one—and formulate the learning problem as a constrained optimisation problem. We evaluate our method on a range of data-sets, including a video-tracking benchmark, where the performance of our unsupervised approach nears that of state-of-the-art supervised approaches, while retaining the computational efficiency of straight-line-based approaches.
BibTeX
@inproceedings{chen2020learning,
  author = {Nutan Chen and Alexej Klushyn and Francesco Ferroni and Justin Bayer and Patrick van der Smagt},
  title = {{Learning Flat Latent Manifolds with VAEs}},
  booktitle = {International Conference on Machine Learning},
  year = {2020},
  url = {https://openreview.net/forum?id=DtYR0hpHWR},
}
Figure