Beyond Gaussian Bottlenecks: Topologically Aligned Encoding of Vision-Transformer Feature Spaces

Andrew Bond, Ilkin Umut Melanlioglu, Erkut Erdem, Aykut ErdemarXiv 2026 · arXiv

TL;DR

Standard Variational Autoencoders (VAEs) rely on Gaussian bottlenecks, creating a fundamental topological mismatch when compressing Vision Transformer (ViT) features, which naturally live on hyperspherical shells. By replacing the Gaussian prior with a product of Power Spherical distributions, $S^2$VAE preserves critical directional and geometric semantics, significantly outperforming traditional VAEs in high-compression regimes for 3D vision tasks.

What Problem Does This Paper Address?

Modern computer vision relies heavily on high-dimensional feature spaces produced by large-scale Vision Transformers (ViTs). To make these features practical for downstream tasks or generative modeling, they must be compressed into a compact latent space using frameworks like Variational Autoencoders (VAEs).

However, standard VAEs assume a Gaussian (Euclidean) latent space. In practice, due to the extensive use of Layer Normalization and dot-product attention, ViT features concentrate on approximately hyperspherical manifolds—thin spherical shells where semantic identity is encoded through angular relationships (directions) rather than Euclidean magnitudes. Forcing this "spherical" data into a dense Euclidean volume creates a severe topological mismatch. This mismatch leads to failure modes like posterior collapse, radial noise that perturbs angular semantics, and gradual semantic drift under strong compression.

Methods

The authors propose $S^2$VAE (Topologically Aligned Encoding of Vision-Transformer Feature Spaces), a geometry-first latent learning framework designed to respect the directional nature of ViT features.

Figure 1. Overview of the proposed S2VAE framework. Input frames are first processed by a frozen ViT backbone (VGGT in this example) to extract multi-layer feature representations. Features from each layer are normalized independently, concatenated, and mapped to a shared hidden space via a gated projection. The encoder then applies a stack of self-attention blocks and predicts the parameters (µ, κ) of a hyperspherical latent distribution, modeled as a product of spheres, from which latent variables z are sampled using spherical reparameterization. These latents are subsequently un-projected, refined through additional self-attention blocks, and projected back to the original feature dimensionality before being split into reconstructed layer features. The reconstructed features can be consumed by the original decoder heads to recover geometry-centric outputs such as camera pose, depth, and 3D point clouds.

Key methodological innovations include:

  • Gated Projections: Before entering the bottleneck, multi-layer ViT features are concatenated and passed through a gated projection layer to suppress redundant geometric information and noise.
  • Product of Spheres: Applying a single high-dimensional spherical distribution causes numerical instability because a sphere's surface area vanishes exponentially as dimensions increase. To solve this, $S^2$VAE decomposes the total latent dimension $D$ into a product of $N$ independent, lower-dimensional sub-spheres (e.g., 16 spheres of 8 dimensions each).
  • Power Spherical Distribution: Instead of the traditional von Mises-Fisher (vMF) distribution, which lacks a closed-form inverse and complicates backpropagation, the model utilizes the Power Spherical distribution. This allows for stable, differentiable sampling via an inverse transform method.
  • Relational Loss Functions: In addition to standard Mean Squared Error (MSE) and cosine similarity reconstruction losses, the authors introduce a loss based on the Gram matrices of the feature layers to preserve inter-feature relational patterns across the scene.

Datasets

The models were trained and evaluated on RealCam-Vid, a diverse video dataset curated from three major sources:

  • RealEstate10K (indoor/outdoor static scenes)
  • DL3DV-10K (metric-scale scans)
  • MiraData (long-duration videos with dynamic camera movements)

This combination provides a robust mix of scene dynamics, camera motions, and environmental types (including real-world and animated/game settings).

Key Insights and Ideas

  • Topological Alignment Saves Capacity: Forcing hyperspherical data into a solid Euclidean ball wastes latent capacity because the model must actively learn to ignore the radial dimension. Aligning the prior's topology with the data's natural manifold removes this burden.
  • Compositional Inductive Bias: Breaking the latent space into a product of multiple spheres allows the model to represent partial semantic overlaps. Tokens can share specific attributes in some sub-spheres while remaining entirely distinct in others.
  • Uniform Dimension Utilization: Gaussian bottlenecks often suffer from "lazy dimensions" where a significant percentage of the latent space goes unused. The spherical bottleneck utilizes 100% of its dimensions uniformly, distributing information more effectively.

Key Results

The authors evaluated $S^2$VAE against conventional Gaussian bottlenecks across several geometry-sensitive downstream tasks using a Visual Geometry Grounded Transformer (VGGT) backbone:

  • Ablation Performance: In a direct bottleneck comparison (Table 3), the product of many Power Spherical distributions achieved an Absolute Relative Error (AbsRel) of 0.04 and a threshold accuracy ($\delta_1$) of 0.98 for depth estimation. The Gaussian bottleneck trailed behind (AbsRel: 0.07, $\delta_1$: 0.95), while a single high-dimensional sphere failed completely due to numerical instability (AbsRel: 1.21).
  • Extreme Compression: $S^2$VAE proved remarkably resilient to heavy compression. Even when reducing the latent space down to an extremely small size of just 8 dimensions, the model still achieved an AbsRel of 0.07 and a $\delta_1$ of 0.95 on depth tasks.
  • Generative Generalization: As a proof of concept, the authors trained a text-conditioned Diffusion Transformer (DiT) directly within the compressed $S^2$VAE latent space. Despite no specific optimization for non-Euclidean spaces, it successfully generated temporally consistent features that accurately rendered into depth maps, camera poses, and point clouds.

Limitations and Future Work

  • Oversmoothing under Heavy Compression: Due to the aggressive compression ratios, the model exhibits a tendency to oversmooth complex textures in reconstructed depth maps, losing some high-frequency details.
  • Non-Optimized Generative Architectures: The downstream DiT experiment relied on standard diffusion components. The authors note that future work will explore diffusion architectures and training strategies natively tailored to hyperspherical latents and other non-Euclidean structures.

Summary updated

Abstract

Modern visual world modeling systems increasingly rely on high-capacity architectures and large-scale data to produce plausible motion, yet they often fail to preserve underlying 3D geometry or physically consistent camera dynamics. A key limitation lies not only in model capacity, but in the latent representations used to encode geometric structure. We propose S$^2$VAE, a geometry-first latent learning framework that focuses on compressing and representing the latent 3D state of a scene, including camera motion, depth, and point-level structure, rather than modeling appearance alone. Building on representations from a Visual Geometry Grounded Transformer (VGGT), we introduce a novel type of variational autoencoder using a product of Power Spherical latent distributions, explicitly enforcing hyperspherical structure in the bottleneck to preserve directional and geometric semantics under strong compression. Across depth estimation, camera pose recovery, and point cloud reconstruction, we show that geometry-aligned hyperspherical latents consistently outperform conventional Gaussian bottlenecks, particularly in high-compression regimes. Our results highlight latent geometry as a first-class design choice for physically grounded visual and world models.
BibTeX
@misc{bond2026beyond,
  author = {Andrew Bond and Ilkin Umut Melanlioglu and Erkut Erdem and Aykut Erdem},
  title = {{Beyond Gaussian Bottlenecks: Topologically Aligned Encoding of Vision-Transformer Feature Spaces}},
  year = {2026},
  eprint = {2604.28122},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2604.28122},
}

Loading related content...

Figure