Episodic looking - using reinforcement learning to bias perception enhances few-shot and continual learning with protection from forgetting

David Rawlinson, Gideon KowadloarXiv 2026 · arXiv

TL;DR

This paper introduces a "Complementary Learning System" (CLS) where a fast-learning Short-Term Memory (STM) uses reinforcement learning to actively bias the internal representations of a slow-learning, frozen Long-Term Memory (LTM). By treating perception as an active, episodic process—essentially "looking" at an image multiple times to refine its interpretation—the model achieves strong few-shot generalization and significant protection against catastrophic forgetting.

What Problem Does This Paper Address?

Artificial neural networks face a fundamental "stability-plasticity" dilemma. To learn new tasks quickly (plasticity), models often overwrite previous knowledge, leading to catastrophic forgetting. Conversely, to remain stable, they often become too rigid to adapt to new, changing data statistics.

Current approaches to continual learning often rely on complex regularization or replaying old data. The authors of this paper propose a different path inspired by cognitive neuroscience: Active Perception. Instead of just trying to classify an input in one shot, can a model learn to "tweak" its own internal perceptual filters to see a novel object in terms it already understands?

Methods

The authors implement a CLS architecture consisting of two main components:

  1. Long-Term Memory (LTM): A standard ResNet-18 pre-trained on a subset of CIFAR-100. Crucially, during the continual learning phase, the LTM is frozen. It provides a stable, generalized feature space.
  2. Short-Term Memory (RL-STM): An Advantage Actor-Critic (A2C) reinforcement learning agent.
Figure 1: Active perception within a Complementary Learning System could allow a STM to learn rapidly without needing to generalize. In this concept, generalization occurs within the LTM, which is manipulated by the STM to emit stable, familiar representations relevant to the task. Using the generalization ability within the LTM, the STM could even immediately reason about completely novel stimuli, if they are expressed in terms the STM can already understand.

The "magic" happens in the interaction loop. For every image, the model performs an 8-step "episode." In each step:

  • The STM observes the LTM’s current internal state and classification logits.
  • The STM outputs a multiplicative bias (a gating signal) that is applied to the LTM’s residual stream (specifically after the second stage).
  • The LTM re-processes the image with this bias, potentially changing the final classification.
  • The STM is rewarded if the bias improves the LTM's classification accuracy.

To prevent the STM itself from forgetting, the authors use Top-K sparsity in its hidden layers, ensuring that only a small subset of neurons is active for any given memory, thereby minimizing representational overlap.

Figure 3: Architecture of the proposed CLS model with a sparse, episodic (RL) Short-Term Memory (STM). The STM is an Advantage Actor-Critic (A2C) model. The actor produces a multiplicative bias output which gates the residual stream of the Long-Term Memory (LTM), a four-stage ResNet- 18 model adapted to CIFAR images. STM input includes previous bias output, post-bias LTM encoding and LTM classification logits. These accumulate in a buffer over each episode; the initial encoding is unbiased. The STM is rewarded for producing bias which improves the accuracy of LTM classification.

Datasets

The study uses the CIFAR-100 dataset, which is organized into 20 "coarse" classes (e.g., aquatic mammals) and 100 "fine" classes (e.g., beaver, dolphin).

  • Pre-training: The LTM is trained on fine-classes 1 and 2 of all 20 coarse classes.
  • Evaluation: The models are tested on their ability to learn the remaining unseen fine-classes (3, 4, and 5) in both few-shot and domain-incremental continual learning settings.

Key Insights and Ideas

  • Perception as an Action: The core "aha" moment is treating the internal bias of a network as a motor action. The STM doesn't just "see"; it "looks" by manipulating the LTM to find a familiar interpretation of a novel stimulus.
  • The "Sweet Spot" for Biasing: The authors found that applying bias to intermediate layers is critical. Early layers are too tied to raw pixels to generalize, while final layers are too invariant to be easily influenced. Gating the residual stream at Stage 2 provided the best balance.
  • Delegated Generalization: Because the STM works in the space of robust perceptions produced by the LTM, it doesn't need to learn how to generalize from scratch. It simply learns which "perceptual levers" to pull to get the right answer from the LTM.
  • Forward Transfer: The RL-STM demonstrated significantly higher Forward Transfer (FWT) than other models. This means that learning one fine-class actually helped the model perform better on subsequent, yet-to-be-seen fine-classes within the same category.

Key Results

  • Protection from Forgetting: In the continual learning setting, the RL-STM was far more resistant to forgetting than a baseline ResNet that was allowed to fine-tune its weights. While the fine-tuned ResNet learned new classes faster, its performance on old classes plummeted.
  • Few-Shot Excellence: For few-shot tasks with $N \ge 16$ samples, the RL-STM outperformed competitive continual learners like the FlyModel and SDMLP.
  • Episodic Uplift: The authors confirmed that the 8-step "looking" process wasn't just noise; accuracy typically improved by about 5% within an episode as the STM refined its bias.
  • Stability-Plasticity Tradeoff: The RL-STM achieved a stability profile similar to specialized models like Nearest Class Mean (NCM) but with better generalization capabilities in few-shot scenarios.

Limitations and Future Work

The primary limitation identified is the representational plasticity of the LTM. Because the LTM is frozen, the system is ultimately limited by the features the LTM learned during pre-training. If the LTM cannot distinguish between two similar-looking classes (like sharks and whales) in its latent space, no amount of STM biasing can perfectly separate them.

The authors suggest that future work should include a consolidation mechanism—a way to occasionally "replay" STM memories to update the LTM's weights without causing catastrophic forgetting, truly completing the biologically-inspired Complementary Learning System loop.

Summary updated

Abstract

There is a fundamental contradiction between the desire for models that can generalize, and models that can continuously adapt to recent input with changing statistics. These conditions, captured in few-shot and continual learning settings, can bias or disrupt learned representations. This harms generalization to a broader context, leaving the model unable to overcome input variance. This paper proposes a solution based on the Complementary Learning System architecture (CLS), in which a Short-Term Memory (STM) delegates generalization to a slow, statistical Long-Term Memory (LTM). Delegation is achieved by working in a space of robust, generalized perceptions produced by the LTM. With this separation, the STM can focus on memorizing recent input and avoiding interference between memories. We show that on an image classification task, the resulting CLS model is more resistant to forgetting in a domain-incremental continual learning setting than a baseline ResNet LTM, and generalizes more effectively in a few-shot setting than other continual learners. In the continual setting, overall accuracy is slightly worse than existing continual learning models, but with a different mix of performance characteristics. Results analysis suggests that the model is limited by the representations and representational plasticity of the LTM. In future work, we will seek to address this by replaying STM memories to the LTM.
BibTeX
@misc{rawlinson2026episodic,
  author = {David Rawlinson and Gideon Kowadlo},
  title = {{Episodic looking - using reinforcement learning to bias perception enhances few-shot and continual learning with protection from forgetting}},
  year = {2026},
  eprint = {2610.38577},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2610.38577},
}
Figure