Brain Inspired Adaptive Memory Dual-Net for Few-Shot Image Classification

Kexin Di, Xiuxing Li, Yuyang Han, Ziyu Li, Qing Li, Xia WuarXiv 2025 · arXiv · OpenReview

Abstract

Few-shot image classification has become a popular research topic for its wide application in real-world scenarios, however the problem of supervision collapse induced by single image-level annotation remains a major challenge. Existing methods aim to tackle this problem by locating and aligning relevant local features. However, the high intra-class variability in real-world images poses significant challenges in locating semantically relevant local regions under few-shot settings. Drawing inspiration from the human's complementary learning system, which excels at rapidly capturing and integrating semantic features from limited examples, we propose the generalization-optimized Systems Consolidation Adaptive Memory Dual-Network, SCAM-Net. This approach simulates the systems consolidation of complementary learning system with an adaptive memory module, which successfully addresses the difficulty of identifying meaningful features in few-shot scenarios. Specifically, we construct a Hippocampus-Neocortex dual-network that consolidates structured representation of each category, the structured representation is then stored and adaptively regulated following the generalization optimization principle in a long-term memory inside Neocortex. Extensive experiments on benchmark datasets show that the proposed model has achieved state-of-the-art performance.

Introduction

Few-shot learning (FSL) represents a significant frontier in computer vision, aiming to train models to recognize new categories from just one or five examples—a feat humans perform effortlessly. However, artificial neural networks often struggle with "supervision collapse," where they over-rely on irrelevant background features or specific training patterns that fail to generalize to novel contexts.

Visualization of supervision collapse and receptive field misalignment in existing few-shot learning methods.

The Challenge: Supervision Collapse and Data Drift

Traditional deep learning models typically require vast amounts of labeled data. In the few-shot regime, two primary issues emerge:

  1. Supervision Collapse: With minimal data, a model might incorrectly identify a "dog" based on the grass in the background rather than the animal's features. Consequently, it fails when encountering a dog in a different environment, such as a beach.
  2. Data Drift: High intra-class variability (e.g., different breeds of dogs) and cluttered backgrounds make it difficult for models to align features between the limited "support" images and new "query" images.

The researchers argue that current AI lacks a biological "complementary learning system" (CLS) that allows humans to rapidly extract semantic features while maintaining a stable, structured understanding of the world.

Methodology: The SCAM-Net Architecture

To address these challenges, the authors propose SCAM-Net (Systems Consolidation Adaptive Memory dual-Network), which mimics the human brain's memory systems through several key innovations:

Dual-Network Architecture

  • Hippocampus Model: This component handles short-term adaptation and rapid learning of episodic information, focusing on the specific task at hand.
  • Neocortex Model: This serves as a long-term repository for structured representations. It is updated slowly using an Exponential Moving Average (EMA) of the Hippocampus's weights, providing a stable anchor that prevents the model from drifting too far during rapid learning.

Generalization-Optimized Consolidation

Unlike standard models that attempt to memorize all available data, SCAM-Net only consolidates memory traces that demonstrably improve generalization. It employs a "consistency loss" to ensure the fast-learning Hippocampus remains aligned with the stable knowledge stored in the Neocortex.

Adaptive Memory Regulation

The Neocortex maintains a long-term memory of global image representations (CLS tokens). When presented with a new task, the model retrieves similar classes from its memory and adaptively adjusts current features. This process effectively "pulls back" deviant features caused by background noise, making the representations more reliable.

Self-Supervised Pre-training

The model utilizes Masked Image Modeling (MIM) during its pre-training phase. By forcing the Vision Transformer (ViT) backbone to reconstruct missing image patches, the model develops a deeper understanding of spatial and semantic structures before it even begins few-shot tasks.

Experimental Results

SCAM-Net was evaluated on four major few-shot classification benchmarks: miniImageNet, tieredImageNet, CIFAR-FS, and FC100.

State-of-the-Art Performance

The model achieved top-tier results across the board, with particularly notable gains in 1-shot scenarios:

  • miniImageNet (1-shot): 75.93% accuracy, outperforming previous benchmarks by over 2%.
  • tieredImageNet (1-shot): 78.89% accuracy.
  • CIFAR-FS (1-shot): 79.45% accuracy.

Computational Efficiency

While the dual-network approach is more intensive during training (2.68 iterations/sec compared to 4.18 for a single network), it requires no additional computational overhead during inference. Only the Neocortex model is utilized for testing, ensuring the model remains practical for real-world deployment.

Conclusion

SCAM-Net demonstrates that brain-inspired memory mechanisms can significantly enhance the robustness of few-shot learning. By treating memory as an adaptive regulator rather than a static storage bin, the model successfully mitigates supervision collapse and feature bias. This "generalization-optimized" principle offers a promising path forward for other domains, including few-shot object detection and continual learning.

Summary updated

Abstract

Few-shot image classification has become a popular research topic for its wide application in real-world scenarios, however the problem of supervision collapse induced by single image-level annotation remains a major challenge. Existing methods aim to tackle this problem by locating and aligning relevant local features. However, the high intra-class variability in real-world images poses significant challenges in locating semantically relevant local regions under few-shot settings. Drawing inspiration from the human's complementary learning system, which excels at rapidly capturing and integrating semantic features from limited examples, we propose the generalization-optimized Systems Consolidation Adaptive Memory Dual-Network, SCAM-Net. This approach simulates the systems consolidation of complementary learning system with an adaptive memory module, which successfully addresses the difficulty of identifying meaningful features in few-shot scenarios. Specifically, we construct a Hippocampus-Neocortex dual-network that consolidates structured representation of each category, the structured representation is then stored and adaptively regulated following the generalization optimization principle in a long-term memory inside Neocortex. Extensive experiments on benchmark datasets show that the proposed model has achieved state-of-the-art performance.
BibTeX
@misc{di2025brain,
  author = {Kexin Di and Xiuxing Li and Yuyang Han and Ziyu Li and Qing Li and Xia Wu},
  title = {{Brain Inspired Adaptive Memory Dual-Net for Few-Shot Image Classification}},
  year = {2025},
  eprint = {2503.07396},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2503.07396},
}
Figure