Superposed Latent Autoencoder

Quanling Zhao, Jiaying Yang, Tianqi Zhang, Ziyang Hao, Fatemeh Asgarinejad, Flavio Ponzina, Tajana RosingarXiv 2026 · arXiv

TL;DR

Instead of shrinking individual latent representations to fit tight memory budgets, the Superposed Latent Autoencoder (SLAE) keeps latents wide and packs multiple representations into a single shared memory tensor. By trading irreversible information loss for structured, recoverable interference, SLAE reduces reconstruction error by up to 56% and boosts downstream classification by up to 16.79 percentage points under the exact same storage budget.

What Problem Does This Paper Address?

Traditional autoencoders compress data by forcing each example through a narrow bottleneck—either by reducing its dimensionality or its spatial resolution. Under tight memory constraints, this standard approach permanently discards crucial information, severely limiting the model's capacity to reconstruct complex data or support downstream tasks.

The authors challenge this paradigm with a clever twist: Does saving memory necessarily require making every representation smaller? Instead of storing each example independently in a shrunken latent space, SLAE encodes examples into wider, high-capacity latents and allows multiple examples to share the same physical storage space through mathematical superposition.

Figure 1: AE vs. SLAE at matched storage. SLAE shares wider latent representations through superposition instead of shrinking each latent independently.

Methods

SLAE introduces a pipeline that decouples the representation used for reconstruction from the representation used for storage:

  1. Encoding: The encoder maps an input into a high-capacity, wide latent representation.
  2. Storage Adaptation: A shallow, identity-initialized residual network transforms this latent into a storage-friendly code without changing its size.
  3. Randomized Binding: Each code is bound to a slot-specific, randomized key using an invertible orthogonal operator (combining signed channel permutations and a Fast Walsh-Hadamard Transform). This step ensures that different latents are unaligned with one another in the high-dimensional space.
  4. Superposition: $K$ bound codes are element-wise added together into a single shared memory tensor.
  5. Recovery & Decoding: To retrieve a specific example, the inverse binding key is applied to the shared memory. This yields the target code mixed with a zero-mean, dimensionally dispersed "structured interference" from the other slots. A learned recovery network (conditioned on the slot ID via FiLM modulation) suppresses this interference, restoring a clean latent for the decoder to reconstruct.

Datasets

The authors evaluated SLAE across five standard image datasets spanning various resolutions and complexities:

  • CIFAR-10 / CIFAR-100
  • SVHN (Street View House Numbers)
  • STL-10
  • Tiny ImageNet

Key Insights and Ideas

  • The Capacity-Interference Tradeoff: Traditional compression creates an irreversible capacity bottleneck. SLAE replaces this with an interference bottleneck. The core philosophy is that it is better to have a rich, wide representation with some recoverable noise than a perfectly clean but severely degraded small representation.
  • The Tight-Budget Sweet Spot: Superposition is highly effective when memory budgets are extremely constrained. In these capacity-limited regimes, the information gained from a wider latent vastly outweighs the difficulty of filtering out cross-slot interference. As the memory budget grows, the relative benefit of extra width diminishes.
Figure 2: Adding latent width (∆d = 8) yields the largest reconstruction gain when the autoencoder is capacity limited, with di- minishing benefit as d/r grows. The dashed line qualitatively separates the two regimes.
  • Complementary to Quantization: SLAE operates on a completely different axis than precision reduction (like INT8 or INT4). Quantization reduces the bits per value, whereas SLAE reduces the number of independently stored tensors. The two can be stacked together for massive storage savings.

Key Results

  • Reconstruction Gains: At matched average storage budgets, SLAE consistently outperforms conventional autoencoders, reducing mean squared error (MSE) by up to 56% in strongly bottlenecked settings.
  • Downstream Utility: The benefits extend far beyond pixel-level reconstruction. When a ResNet-18 classifier was trained from scratch on images reconstructed by SLAE (at a budget of $B=64$ scalars/image), downstream top-1 accuracy on CIFAR-100 improved by 16.79 percentage points compared to a standard autoencoder.
  • Synergy with Quantization: At an 8× compression ratio on CIFAR-10, combining $K=2$ superposition with INT8 quantization achieved a 37.3% lower MSE than using aggressive INT4 quantization alone under the same bit budget.

Limitations and Future Work

  • The High-$K$ Regime: While increasing the number of superposed examples ($K$) provides access to wider latents, it also exponentially increases the recovery penalty. In practice, $K=2$ proved to be the most reliable operating point, while $K=4$ and $K=8$ were only beneficial in highly specific settings.
  • Parameter Overhead: The current recovery network architecture scales with the channel dimension, making it parameter-expensive when dealing with exceptionally wide latents.
  • Fixed Superposition Factor: Each value of $K$ must currently be trained separately. The model cannot dynamically adjust its degree of superposition at deployment time based on fluctuating memory availability.

Future directions highlighted by the authors include developing more parameter-efficient recovery modules, exploring adaptive grouping strategies, and extending the superposition principle to language model KV caches and vector databases.

Summary updated

Abstract

Autoencoders typically meet tight latent-memory budgets by making each latent representation smaller, sacrificing representational capacity. We ask a different question: can multiple wider latents be stored together instead? We introduce the Superposed Latent Autoencoder (SLAE), which preserves high-capacity latent representations while sharing storage through learned superposition. SLAE transforms latents into storage-friendly codes, binds them with randomized keys, superposes multiple codes into a single memory tensor, and learns to recover each latent before decoding. Under the same storage budget, SLAE replaces irreversible dimensional bottlenecks with structured interference that can be suppressed. Across CIFAR-10/100, SVHN, STL-10, Tiny ImageNet, and a wide range of memory budgets, SLAE substantially improves the reconstruction--memory tradeoff, reducing reconstruction error by up to 56% over conventional autoencoders at matched storage. Further analysis shows that SLAE's advantage comes from making wider representations usable under the same storage budget. These gains also extend beyond reconstruction: the information preserved by SLAE improves downstream classification by up to 16.79 percentage points under the same memory budget. Our results suggest a new principle for representation compression: instead of making every latent smaller, keep representations wide and let them share memory.
BibTeX
@misc{zhao2026superposed,
  author = {Quanling Zhao and Jiaying Yang and Tianqi Zhang and Ziyang Hao and Fatemeh Asgarinejad and Flavio Ponzina and Tajana Rosing},
  title = {{Superposed Latent Autoencoder}},
  year = {2026},
  eprint = {2609.01158},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2609.01158},
}
Figure