Abstract
Measuring the similarity between data points often requires domain knowledge, which can in parts be compensated by relying on unsupervised methods such as latent-variable models, where similarity/distance is estimated in a more compact latent space. Prevalent is the use of the Euclidean metric, which has the drawback of ignoring information about similarity of data stored in the decoder, as captured by the framework of Riemannian geometry. We propose an extension to the framework of variational auto-encoders allows learning flat latent manifolds, where the Euclidean metric is a proxy for the similarity between data points. This is achieved by defining the latent space as a Riemannian manifold and by regularising the metric tensor to be a scaled identity matrix. Additionally, we replace the compact prior typically used in variational auto-encoders with a recently presented, more expressive hierarchical one---and formulate the learning problem as a constrained optimisation problem. We evaluate our method on a range of data-sets, including a video-tracking benchmark, where the performance of our unsupervised approach nears that of state-of-the-art supervised approaches, while retaining the computational efficiency of straight-line-based approaches.
Introduction
In machine learning, measuring similarity between high-dimensional data points like images or video frames is essential for tasks such as clustering, tracking, and retrieval. Latent variable models, specifically Variational Autoencoders (VAEs), compress data into a lower-dimensional space where Euclidean distance is often assumed to represent meaningful similarity. However, standard VAEs distort this space through non-linear decoders, meaning straight lines in latent space do not correspond to the shortest paths (geodesics) on the underlying data manifold. Computing these geodesics is computationally expensive and impractical for real-time applications.
Methods: The Flat Manifold VAE (FMVAE)
The proposed Flat Manifold VAE (FMVAE) forces the latent space to be isometric to Euclidean space through three key innovations:
- Hierarchical Prior: Utilizing a Variational Hierarchical Prior (VHP) instead of a simple Gaussian prior allows for more expressive, non-compact latent representations.
- Jacobian Regularization: A regularization term is added to the loss function to force the Riemannian metric tensor—derived from the decoder's Jacobian—to approximate a scaled identity matrix ($c^2 \mathbf{I}$). A fast stochastic approximation is used to maintain computational efficiency.
- Latent Mixup: To ensure the manifold is flat even in regions without training data, the authors apply mixup data augmentation in the latent space. By regularizing synthetic points between encoded data, the model learns a globally flat geometry.
Key Results
The VHP-FMVAE was evaluated on several datasets, including an artificial pendulum, MNIST, and the CMU Human Motion Capture database.
Geometric Flatness
Quantitative evaluation shows that the VHP-FMVAE successfully flattens the latent manifold. The proposed method achieves a condition number and normalized Magnification Factor (MF) significantly closer to 1.0 than the baseline VHP-VAE, indicating a nearly constant metric tensor and a lack of distortion.
Smooth Interpolations and Tracking
For human motion data, the ratio of Euclidean interpolation length to true geodesic length was $1.02 \pm 0.06$, nearly perfect compared to the $1.23 \pm 0.20$ of the baseline. This geometric alignment translates to smoother animations during interpolation without unnatural jumps.
In real-world tracking on the MOT16 benchmark, the unsupervised VHP-FMVAE achieved an IDF1 score of 64.2 when used as a descriptor for the SORT algorithm. This performance drastically outperforms standard VAE-SORT (60.5) and nearly matches the fully supervised DeepSORT baseline (64.7), demonstrating that a geometrically sound unsupervised space can compete with heavily engineered supervised metric learning.
Conclusion
The VHP-FMVAE bridges the gap between the accuracy of Riemannian geometry and the speed of Euclidean metrics. By explicitly regularizing the latent space's geometry, the model enables high-quality similarity matching without the need for manual labels. Future work may involve scaling this regularization to other generative architectures like GANs or diffusion models for more complex real-world datasets.
Summary updated