Abstract: We present a method for storing multiple models within a single set of parameters. Models can coexist in superposition and still be retrieved individually. In experiments with neural networks, we show that a surprisingly large number of models can be effectively stored within a single parameter instance. Furthermore, each of these models can undergo thousands of training steps without significantly interfering with other models within the superposition. This approach may be viewed as the online complement of compression: rather than reducing the size of a network after training, we make use of the unrealized capacity of a network during training.
Introduction
Researchers from UC Berkeley have developed a method called Parameter Superposition (PSP) that allows a single neural network to store and retrieve multiple distinct models within the same set of weights. By using task-specific "context" keys to rotate data into nearly orthogonal subspaces, they effectively eliminate catastrophic forgetting, allowing a single network to learn dozens of tasks sequentially with minimal performance loss.
The Problem: Catastrophic Forgetting
Deep neural networks are notoriously "over-parameterized"—they possess far more weights than strictly necessary to solve a single task. Usually, this redundancy is only addressed post-training through pruning. However, if one attempts to teach a network a second task after the first, it typically suffers from catastrophic forgetting: new gradients overwrite old ones, erasing previous knowledge.
Existing solutions like Elastic Weight Consolidation (EWC) attempt to slow changes to "important" weights, but these methods eventually hit a capacity ceiling. PSP asks: Can we use that "wasted" capacity during training to pack multiple models into one, without them interfering with each other?
Methods: Parameter Superposition (PSP)
The core idea of PSP is to treat the network's weights as a "memory" and use task-specific "context" vectors as "keys" to access that memory.
- The Superposition Principle: Inspired by Fourier analysis, PSP stores weights for $K$ tasks ($W_1, W_2, ... W_K$) by summing them into a single parameter instance $W$.
- Contextual Routing: To prevent models from interfering, each task is assigned a context $C_k$ (a linear transformation). When performing task $k$, the input is transformed by this context, effectively "rotating" it into a specific subspace where only the relevant model parameters are active.
- Types of Context: The authors explored rotational (orthogonal matrices), complex (unit circle values), and binary contexts. Binary contexts ($\{-1, 1\}$) proved most practical, requiring only one additional bit per weight per task.
- Extension to CNNs: For convolutional layers, contexts are applied directly to the kernels rather than input images for computational efficiency.
Key Insights
- Exploiting Low Dimensionality: Natural data (like images) lives on low-dimensional manifolds. Because data doesn't fill the entire "ambient space" of the network, different tasks can be rotated into different "corners" of that space without collision.
- Online Compression: Instead of post-hoc compression, PSP acts as "online compression," filling the network's unrealized capacity during the learning process.
- Interference as Noise: When retrieving a model, other models act as unbiased noise. The paper mathematically demonstrates that with correctly chosen contexts, this noise has low variance, allowing for accurate inference.
Key Results
PSP was evaluated on benchmarks including Permuting MNIST, Rotating MNIST, and Incremental CIFAR (iCIFAR) using architectures like ResNet-18.
- Superiority over Baselines: On Permuting MNIST, PSP maintained ~97% accuracy after 50 tasks, while standard networks dropped to ~60%. It significantly outperformed established methods like EWC and Synaptic Intelligence (SI).
- ResNet Scaling: PSP successfully scaled to ResNet-18 on iCIFAR. While a standard ResNet's performance on the first task plummeted as it learned new ones, the PSP-enabled version showed virtually no degradation.
- Efficiency: The binary version of the method is highly memory-efficient, often requiring minimal additional parameters to maintain the entire model's state across many tasks.
Limitations and Future Work
Currently, PSP requires task identity to be provided to select the correct context key. Future work aims to determine contexts automatically from input data. Additionally, while PSP fits 50+ tasks into one model, a formal analysis of the absolute "superposition capacity"—the point where noise becomes overwhelming—remains an open research question.
Summary updated