TL;DR
Instead of shrinking individual latent representations to fit tight memory budgets, the Superposed Latent Autoencoder (SLAE) keeps latents wide and packs multiple representations into a single shared memory tensor. By trading irreversible information loss for structured, recoverable interference, SLAE reduces reconstruction error by up to 56% and boosts downstream classification by up to 16.79 percentage points under the exact same storage budget.
What Problem Does This Paper Address?
Traditional autoencoders compress data by forcing each example through a narrow bottleneck—either by reducing its dimensionality or its spatial resolution. Under tight memory constraints, this standard approach permanently discards crucial information, severely limiting the model's capacity to reconstruct complex data or support downstream tasks.
The authors challenge this paradigm with a clever twist: Does saving memory necessarily require making every representation smaller? Instead of storing each example independently in a shrunken latent space, SLAE encodes examples into wider, high-capacity latents and allows multiple examples to share the same physical storage space through mathematical superposition.
Methods
SLAE introduces a pipeline that decouples the representation used for reconstruction from the representation used for storage:
- Encoding: The encoder maps an input into a high-capacity, wide latent representation.
- Storage Adaptation: A shallow, identity-initialized residual network transforms this latent into a storage-friendly code without changing its size.
- Randomized Binding: Each code is bound to a slot-specific, randomized key using an invertible orthogonal operator (combining signed channel permutations and a Fast Walsh-Hadamard Transform). This step ensures that different latents are unaligned with one another in the high-dimensional space.
- Superposition: $K$ bound codes are element-wise added together into a single shared memory tensor.
- Recovery & Decoding: To retrieve a specific example, the inverse binding key is applied to the shared memory. This yields the target code mixed with a zero-mean, dimensionally dispersed "structured interference" from the other slots. A learned recovery network (conditioned on the slot ID via FiLM modulation) suppresses this interference, restoring a clean latent for the decoder to reconstruct.
Datasets
The authors evaluated SLAE across five standard image datasets spanning various resolutions and complexities:
- CIFAR-10 / CIFAR-100
- SVHN (Street View House Numbers)
- STL-10
- Tiny ImageNet
Key Insights and Ideas
- The Capacity-Interference Tradeoff: Traditional compression creates an irreversible capacity bottleneck. SLAE replaces this with an interference bottleneck. The core philosophy is that it is better to have a rich, wide representation with some recoverable noise than a perfectly clean but severely degraded small representation.
- The Tight-Budget Sweet Spot: Superposition is highly effective when memory budgets are extremely constrained. In these capacity-limited regimes, the information gained from a wider latent vastly outweighs the difficulty of filtering out cross-slot interference. As the memory budget grows, the relative benefit of extra width diminishes.
- Complementary to Quantization: SLAE operates on a completely different axis than precision reduction (like INT8 or INT4). Quantization reduces the bits per value, whereas SLAE reduces the number of independently stored tensors. The two can be stacked together for massive storage savings.
Key Results
- Reconstruction Gains: At matched average storage budgets, SLAE consistently outperforms conventional autoencoders, reducing mean squared error (MSE) by up to 56% in strongly bottlenecked settings.
- Downstream Utility: The benefits extend far beyond pixel-level reconstruction. When a ResNet-18 classifier was trained from scratch on images reconstructed by SLAE (at a budget of $B=64$ scalars/image), downstream top-1 accuracy on CIFAR-100 improved by 16.79 percentage points compared to a standard autoencoder.
- Synergy with Quantization: At an 8× compression ratio on CIFAR-10, combining $K=2$ superposition with INT8 quantization achieved a 37.3% lower MSE than using aggressive INT4 quantization alone under the same bit budget.
Limitations and Future Work
- The High-$K$ Regime: While increasing the number of superposed examples ($K$) provides access to wider latents, it also exponentially increases the recovery penalty. In practice, $K=2$ proved to be the most reliable operating point, while $K=4$ and $K=8$ were only beneficial in highly specific settings.
- Parameter Overhead: The current recovery network architecture scales with the channel dimension, making it parameter-expensive when dealing with exceptionally wide latents.
- Fixed Superposition Factor: Each value of $K$ must currently be trained separately. The model cannot dynamically adjust its degree of superposition at deployment time based on fluctuating memory availability.
Future directions highlighted by the authors include developing more parameter-efficient recovery modules, exploring adaptive grouping strategies, and extending the superposition principle to language model KV caches and vector databases.
Summary updated