TL;DR
This paper introduces a "Complementary Learning System" (CLS) where a fast-learning Short-Term Memory (STM) uses reinforcement learning to actively bias the internal representations of a slow-learning, frozen Long-Term Memory (LTM). By treating perception as an active, episodic process—essentially "looking" at an image multiple times to refine its interpretation—the model achieves strong few-shot generalization and significant protection against catastrophic forgetting.
What Problem Does This Paper Address?
Artificial neural networks face a fundamental "stability-plasticity" dilemma. To learn new tasks quickly (plasticity), models often overwrite previous knowledge, leading to catastrophic forgetting. Conversely, to remain stable, they often become too rigid to adapt to new, changing data statistics.
Current approaches to continual learning often rely on complex regularization or replaying old data. The authors of this paper propose a different path inspired by cognitive neuroscience: Active Perception. Instead of just trying to classify an input in one shot, can a model learn to "tweak" its own internal perceptual filters to see a novel object in terms it already understands?
Methods
The authors implement a CLS architecture consisting of two main components:
- Long-Term Memory (LTM): A standard ResNet-18 pre-trained on a subset of CIFAR-100. Crucially, during the continual learning phase, the LTM is frozen. It provides a stable, generalized feature space.
- Short-Term Memory (RL-STM): An Advantage Actor-Critic (A2C) reinforcement learning agent.
The "magic" happens in the interaction loop. For every image, the model performs an 8-step "episode." In each step:
- The STM observes the LTM’s current internal state and classification logits.
- The STM outputs a multiplicative bias (a gating signal) that is applied to the LTM’s residual stream (specifically after the second stage).
- The LTM re-processes the image with this bias, potentially changing the final classification.
- The STM is rewarded if the bias improves the LTM's classification accuracy.
To prevent the STM itself from forgetting, the authors use Top-K sparsity in its hidden layers, ensuring that only a small subset of neurons is active for any given memory, thereby minimizing representational overlap.
Datasets
The study uses the CIFAR-100 dataset, which is organized into 20 "coarse" classes (e.g., aquatic mammals) and 100 "fine" classes (e.g., beaver, dolphin).
- Pre-training: The LTM is trained on fine-classes 1 and 2 of all 20 coarse classes.
- Evaluation: The models are tested on their ability to learn the remaining unseen fine-classes (3, 4, and 5) in both few-shot and domain-incremental continual learning settings.
Key Insights and Ideas
- Perception as an Action: The core "aha" moment is treating the internal bias of a network as a motor action. The STM doesn't just "see"; it "looks" by manipulating the LTM to find a familiar interpretation of a novel stimulus.
- The "Sweet Spot" for Biasing: The authors found that applying bias to intermediate layers is critical. Early layers are too tied to raw pixels to generalize, while final layers are too invariant to be easily influenced. Gating the residual stream at Stage 2 provided the best balance.
- Delegated Generalization: Because the STM works in the space of robust perceptions produced by the LTM, it doesn't need to learn how to generalize from scratch. It simply learns which "perceptual levers" to pull to get the right answer from the LTM.
- Forward Transfer: The RL-STM demonstrated significantly higher Forward Transfer (FWT) than other models. This means that learning one fine-class actually helped the model perform better on subsequent, yet-to-be-seen fine-classes within the same category.
Key Results
- Protection from Forgetting: In the continual learning setting, the RL-STM was far more resistant to forgetting than a baseline ResNet that was allowed to fine-tune its weights. While the fine-tuned ResNet learned new classes faster, its performance on old classes plummeted.
- Few-Shot Excellence: For few-shot tasks with $N \ge 16$ samples, the RL-STM outperformed competitive continual learners like the FlyModel and SDMLP.
- Episodic Uplift: The authors confirmed that the 8-step "looking" process wasn't just noise; accuracy typically improved by about 5% within an episode as the STM refined its bias.
- Stability-Plasticity Tradeoff: The RL-STM achieved a stability profile similar to specialized models like Nearest Class Mean (NCM) but with better generalization capabilities in few-shot scenarios.
Limitations and Future Work
The primary limitation identified is the representational plasticity of the LTM. Because the LTM is frozen, the system is ultimately limited by the features the LTM learned during pre-training. If the LTM cannot distinguish between two similar-looking classes (like sharks and whales) in its latent space, no amount of STM biasing can perfectly separate them.
The authors suggest that future work should include a consolidation mechanism—a way to occasionally "replay" STM memories to update the LTM's weights without causing catastrophic forgetting, truly completing the biologically-inspired Complementary Learning System loop.
Summary updated