Overview
Instead of pruning over-parameterized neural networks after training, why not use that "wasted" capacity to store dozens of different models simultaneously? This paper introduces Parameter Superposition (PSP), a method that allows multiple task-specific models to coexist within a single set of weights, effectively neutralizing catastrophic forgetting by treating the network as a high-capacity associative memory.
What Problem Does This Paper Address?
Deep neural networks are notoriously "wasteful." We know this because we can often prune away 90% of a trained network's weights without losing accuracy. However, we haven't been great at using that extra 90% during training.
The biggest hurdle is catastrophic forgetting: when a model learns a new task, it overwrites the weights used for previous tasks, causing its performance on the old data to plummet. Existing solutions usually involve "freezing" important weights (which limits future learning) or keeping "replay buffers" of old data (which is memory-intensive). The authors ask: Can we exploit a network's inherent redundancy to learn many tasks at once without them interfering with each other?
Methods
The core idea is Parameter Superposition (PSP). Think of the network's weights ($W$) as a shared memory. For each task $k$, we define a unique "context" ($C_k$)—essentially a mathematical key.
- Storage: During training, the weights for task $k$ are transformed by the inverse context ($C_k^{-1}$) and added to the total weight pool.
- Retrieval: To perform task $k$, we apply the context $C_k$ to the inputs (or weights) to "extract" the relevant model from the superposition.
The authors explore several ways to define these contexts:
- Rotational Superposition: Using orthogonal matrices to rotate the data into different subspaces.
- Binary Superposition: A memory-efficient version using diagonal matrices with entries of $\pm 1$. This is the "easiest to implement" version.
- Complex Superposition: Using complex numbers (phases) on the unit circle.
- Convolutional Extension: For CNNs, they apply the context to the convolutional kernels rather than the input images, which is computationally more efficient.
Datasets
The researchers evaluated PSP across several challenging benchmarks:
- Permuted MNIST: A classic catastrophic forgetting benchmark where pixels are randomly shuffled for each new task.
- iCIFAR (Incremental CIFAR): The model learns CIFAR-10 first, then sequentially learns disjoint sets of classes from CIFAR-100.
- Rotating MNIST & Rotating FashionMNIST: New benchmarks introduced in this paper to simulate "continuous domain shift." Instead of discrete tasks, the images rotate slowly over time (like day turning to night), requiring the model to adapt to a smoothly changing distribution.
Key Insights and Ideas
- The "Online Complement" of Compression: Usually, we compress models after they are done. PSP is "online" compression—it uses the unrealized capacity of the network while it is learning.
- Interference is Minimal: You might expect 50 models in one set of weights to be a noisy mess. However, because natural data (like images) is intrinsically low-dimensional, there is plenty of "empty space" in the high-dimensional weight matrix to store these models with negligible interference.
- Capacity Scales with Size: As networks get larger, their ability to store models in superposition improves significantly.
In a 2048-unit network, the performance on the first task remained virtually unchanged even after learning 49 other tasks, whereas smaller networks showed more degradation.
- Context as Keys: The authors draw a beautiful parallel to Kanerva’s hetero-associative memory, where the weights are the memory and the task-specific contexts are the keys used to access it.
Key Results
- Beating the Baselines: On Permuted MNIST (50 tasks), Binary PSP achieved 97.6% average accuracy, outperforming established methods like Elastic Weight Consolidation (EWC, 97.0%) and Synaptic Intelligence (SI, 97.2%). A standard "vanilla" network dropped to 61.8%.
- Robustness to Output Shift: In the iCIFAR experiment, a standard ResNet-18 crashed when the output labels changed. The PSP-enhanced ResNet-18 was "virtually unaffected," maintaining high accuracy on the original CIFAR-10 task despite learning four subsequent tasks.
- Handling Continuous Change: PSP proved highly effective at the Rotating MNIST task, showing that it can handle slow, real-world-style transitions if provided with the right temporal context.
Limitations and Future Work
- The "Task Identity" Requirement: Currently, the model needs to be told which task it is doing so it can use the correct context "key." The authors acknowledge that a truly autonomous system should be able to determine the context dynamically from the input data.
- Upper Limits: The paper shows we can store dozens of models, but it doesn't define the theoretical "breaking point" where the superposition becomes too noisy to function.
- Differentiable Context: A promising future direction mentioned is making the context vectors differentiable, allowing the network to learn the best way to partition its own memory.
Summary updated