TL;DR
Standard Variational Autoencoders (VAEs) rely on Gaussian bottlenecks, creating a fundamental topological mismatch when compressing Vision Transformer (ViT) features, which naturally live on hyperspherical shells. By replacing the Gaussian prior with a product of Power Spherical distributions, $S^2$VAE preserves critical directional and geometric semantics, significantly outperforming traditional VAEs in high-compression regimes for 3D vision tasks.
What Problem Does This Paper Address?
Modern computer vision relies heavily on high-dimensional feature spaces produced by large-scale Vision Transformers (ViTs). To make these features practical for downstream tasks or generative modeling, they must be compressed into a compact latent space using frameworks like Variational Autoencoders (VAEs).
However, standard VAEs assume a Gaussian (Euclidean) latent space. In practice, due to the extensive use of Layer Normalization and dot-product attention, ViT features concentrate on approximately hyperspherical manifolds—thin spherical shells where semantic identity is encoded through angular relationships (directions) rather than Euclidean magnitudes. Forcing this "spherical" data into a dense Euclidean volume creates a severe topological mismatch. This mismatch leads to failure modes like posterior collapse, radial noise that perturbs angular semantics, and gradual semantic drift under strong compression.
Methods
The authors propose $S^2$VAE (Topologically Aligned Encoding of Vision-Transformer Feature Spaces), a geometry-first latent learning framework designed to respect the directional nature of ViT features.
Key methodological innovations include:
- Gated Projections: Before entering the bottleneck, multi-layer ViT features are concatenated and passed through a gated projection layer to suppress redundant geometric information and noise.
- Product of Spheres: Applying a single high-dimensional spherical distribution causes numerical instability because a sphere's surface area vanishes exponentially as dimensions increase. To solve this, $S^2$VAE decomposes the total latent dimension $D$ into a product of $N$ independent, lower-dimensional sub-spheres (e.g., 16 spheres of 8 dimensions each).
- Power Spherical Distribution: Instead of the traditional von Mises-Fisher (vMF) distribution, which lacks a closed-form inverse and complicates backpropagation, the model utilizes the Power Spherical distribution. This allows for stable, differentiable sampling via an inverse transform method.
- Relational Loss Functions: In addition to standard Mean Squared Error (MSE) and cosine similarity reconstruction losses, the authors introduce a loss based on the Gram matrices of the feature layers to preserve inter-feature relational patterns across the scene.
Datasets
The models were trained and evaluated on RealCam-Vid, a diverse video dataset curated from three major sources:
- RealEstate10K (indoor/outdoor static scenes)
- DL3DV-10K (metric-scale scans)
- MiraData (long-duration videos with dynamic camera movements)
This combination provides a robust mix of scene dynamics, camera motions, and environmental types (including real-world and animated/game settings).
Key Insights and Ideas
- Topological Alignment Saves Capacity: Forcing hyperspherical data into a solid Euclidean ball wastes latent capacity because the model must actively learn to ignore the radial dimension. Aligning the prior's topology with the data's natural manifold removes this burden.
- Compositional Inductive Bias: Breaking the latent space into a product of multiple spheres allows the model to represent partial semantic overlaps. Tokens can share specific attributes in some sub-spheres while remaining entirely distinct in others.
- Uniform Dimension Utilization: Gaussian bottlenecks often suffer from "lazy dimensions" where a significant percentage of the latent space goes unused. The spherical bottleneck utilizes 100% of its dimensions uniformly, distributing information more effectively.
Key Results
The authors evaluated $S^2$VAE against conventional Gaussian bottlenecks across several geometry-sensitive downstream tasks using a Visual Geometry Grounded Transformer (VGGT) backbone:
- Ablation Performance: In a direct bottleneck comparison (Table 3), the product of many Power Spherical distributions achieved an Absolute Relative Error (AbsRel) of 0.04 and a threshold accuracy ($\delta_1$) of 0.98 for depth estimation. The Gaussian bottleneck trailed behind (AbsRel: 0.07, $\delta_1$: 0.95), while a single high-dimensional sphere failed completely due to numerical instability (AbsRel: 1.21).
- Extreme Compression: $S^2$VAE proved remarkably resilient to heavy compression. Even when reducing the latent space down to an extremely small size of just 8 dimensions, the model still achieved an AbsRel of 0.07 and a $\delta_1$ of 0.95 on depth tasks.
- Generative Generalization: As a proof of concept, the authors trained a text-conditioned Diffusion Transformer (DiT) directly within the compressed $S^2$VAE latent space. Despite no specific optimization for non-Euclidean spaces, it successfully generated temporally consistent features that accurately rendered into depth maps, camera poses, and point clouds.
Limitations and Future Work
- Oversmoothing under Heavy Compression: Due to the aggressive compression ratios, the model exhibits a tendency to oversmooth complex textures in reconstructed depth maps, losing some high-frequency details.
- Non-Optimized Generative Architectures: The downstream DiT experiment relied on standard diffusion components. The authors note that future work will explore diffusion architectures and training strategies natively tailored to hyperspherical latents and other non-Euclidean structures.
Summary updated