Abstract
This paper introduces Attentive Independent Mechanisms (AIM), a modular architectural component designed to address the stability-plasticity dilemma in deep neural networks. By employing competitive attention among independent experts, AIM facilitates sparse updates that protect existing knowledge while enabling rapid adaptation to new tasks. Integrating AIM into established meta-learning frameworks yields state-of-the-art results in few-shot classification and significantly reduces catastrophic forgetting in continual learning scenarios.
Introduction
Deep learning models often struggle with the stability-plasticity dilemma: the trade-off between retaining old knowledge (stability) and acquiring new information (plasticity). In few-shot learning, models must adapt quickly from minimal data, while in continual learning, they must learn sequences of tasks without overwriting previous ones. Current architectures often lack the structural inductive bias to decouple general feature extraction from task-specific reasoning. AIM addresses this by forcing modularity through competition, ensuring that new information is integrated without destroying previously learned concepts.
Methodology
The core innovation is the AIM module, a static adaptation of Recurrent Independent Mechanisms (RIMs). AIM consists of $M$ independent mechanisms (experts) that compete for activation via a cross-attention mechanism.
For any given input, only the top $K$ experts are activated. This sparsity is crucial: because only a few experts are active during a forward pass, only their weights are updated during backpropagation. This prevents new information from "overwriting" the knowledge stored in inactive experts. AIM is typically positioned between a "Representation Learning Network" (slow weights) and a "Prediction Learning Network" (fast weights). To prevent the model from relying on a small subset of mechanisms (mechanism-overfitting), the authors employ a stochastic intervention strategy during training, forcing the exploration of different expert combinations.
Key Insights and Results
The authors evaluated AIM across standard benchmarks including MiniImageNet, CIFAR-FS, and Omniglot. When integrated into the Synthetic Information Bottleneck (SIB) framework, AIM achieved a new benchmark of 71.22% accuracy on 5-way, 1-shot MiniImageNet. The impact on continual learning was even more pronounced, with AIM improving the Online Meta-Learning (OML) baseline by a relative 20.7% on the Omniglot dataset after 600 classes.
Qualitative analysis of attention weights (see above) reveals that mechanisms specialize: some act as "generalists" active across many classes, while others become "specialists" for specific features. The authors demonstrated that this success is due to the competitive structure rather than simple parameter increases, as baseline models with equivalent parameter counts but standard linear layers performed significantly worse.
Conclusion and Future Work
AIM provides a robust structural bias for modular learning, effectively mitigating catastrophic forgetting through isolated knowledge buffers. While highly effective, the performance remains sensitive to the choice of $K$ (active experts). Future work will explore stacking multiple AIM layers to enable hierarchical concept modeling, potentially capturing low-level features and high-level objects in separate modular stages.
Summary updated