Meta-Learning via Learning with Distributed Memory

Sudarshan Babu, Pedro Savarese, Michael MaireNeurIPS 2021 · OpenReview

TL;DR: This paper simplifies meta-learning by replacing complex bi-level optimization (like MAML) with standard end-to-end training of deep networks containing "layer-distributed memory." By embedding Convolutional LSTM cells into every layer and injecting label feedback directly into the architecture, the model learns to adapt its internal activations in real-time, outperforming traditional methods in online few-shot and continual learning scenarios.

The Challenge of Meta-Learning

Meta-learning, or "learning to learn," aims to create models that can adapt to new tasks with only a few examples. Traditionally, this is solved in two ways:

  1. Optimization-based (e.g., MAML): These use a "nested" optimization where an inner loop adapts to a specific task and an outer loop learns the initialization. This is computationally expensive and often unstable.
  2. Memory-based (e.g., Prototypes): These store specific examples or "prototypes" to compare against new data. While efficient, they are often rigid, relying on fixed representation sizes and specific "read/write" protocols.

The authors identify a specific gap: most existing methods struggle with online learning, especially when there is a delay between seeing a sample and receiving its label. They propose that instead of treating adaptation as a separate optimization step, it should be an implicit function of the network's persistent internal state (memory) distributed across all layers.

The Distributed Memory Architecture

The core contribution is a shift in architecture and training philosophy:

  • Distributed Memory: Instead of having a single memory module at the end of a feature extractor, the authors replace standard convolutional filters with Convolutional LSTMs (ConvLSTM) throughout the entire depth of the network.
  • Adaptation in Activation Space: The model views the LSTM hidden state as a way to modulate activations. Every layer is responsible for generating hidden states that help the next time step's prediction.
  • Label Injection: To guide adaptation, the ground-truth label from the previous time step is fed back into the network. Crucially, injecting this label into every layer (rather than just the last one) significantly boosts performance, allowing the entire hierarchy of features to adapt to the current task.
Architecture diagram showing ConvLSTM layers and label injection

Because adaptation is handled entirely by the recurrent states, the model can be trained using standard Backpropagation Through Time (BPTT) on entire episodes, treating meta-learning as a simple supervised sequence modeling problem.

Key Insights

  • The Power of Distribution: Adaptation isn't just for the final classifier. By distributing memory, the model can adjust low-level textures and mid-level shapes to suit a specific task, not just the final category mapping.
  • Handling Delays Naturally: Because the model relies on a persistent recurrent state, it handles "delayed feedback" (where the label arrives several steps late) much better than gradient-based methods. It doesn't need to store a buffer of old images; the "memory" of the image is already encoded in the hidden states.
  • Efficiency at Inference: During the adaptation phase (inference), the model requires no gradient computations. It simply performs a forward pass, making it much faster and more energy-efficient for edge devices than MAML-like approaches.

Key Results

The researchers evaluated their approach across several challenging benchmarks, including few-shot classification (Omniglot, CIFAR-FS), online continual learning, and online few-shot semantic segmentation (FSS-1000).

Table comparing few-shot classification results on Omniglot and CIFAR-FS
  • Omniglot & CIFAR-FS: The CL+LSTM model achieved 99.7% accuracy on 5-way 8-shot online learning for Omniglot, and reached 61.0% on 8-shot tasks for CIFAR-FS, showing a significant jump over baselines like Contextual Prototypical Memory (CPM) and MAML.
  • Delayed Feedback: In a 5-way CIFAR-FS task with a 5-step label delay, the model maintained 57.4% accuracy, whereas other methods that require immediate labels would fail or require massive external buffers.
  • Segmentation: The "CL U-Net" variant was the only model tested that could successfully "mask out" distractor objects in a few-shot segmentation stream, proving the memory effectively learns the "task" (what to segment) rather than just memorizing labels.

Limitations and Future Work

While inference is fast, training requires BPTT over long sequences, which can be memory-intensive and prone to vanishing/exploding gradients. Furthermore, LSTMs are generally better at short-to-medium term memory. For "life-long" learning scenarios spanning thousands of steps, this architecture might need to be combined with external memory or attention mechanisms. Finally, scaling this "distributed memory" concept from smaller CNNs to massive models like Vision Transformers remains an open engineering challenge.

Summary updated

Abstract

We demonstrate that efficient meta-learning can be achieved via end-to-end training of deep neural networks with memory distributed across layers. The persistent state of this memory assumes the entire burden of guiding task adaptation. Moreover, its distributed nature is instrumental in orchestrating adaptation. Ablation experiments demonstrate that providing relevant feedback to memory units distributed across the depth of the network enables them to guide adaptation throughout the entire network. Our results show that this is a successful strategy for simplifying meta-learning -- often cast as a bi-level optimization problem -- to standard end-to-end training, while outperforming gradient-based, prototype-based, and other memory-based meta-learning strategies. Additionally, our adaptation strategy naturally handles online learning scenarios with a significant delay between observing a sample and its corresponding label -- a setting in which other approaches struggle. Adaptation via distributed memory is effective across a wide range of learning tasks, ranging from classification to online few-shot semantic segmentation.
BibTeX
@inproceedings{babu2021meta,
  author = {Sudarshan Babu and Pedro Savarese and Michael Maire},
  title = {{Meta-Learning via Learning with Distributed Memory}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2021},
  url = {https://openreview.net/forum?id=SDjLEFoO_33},
}
Figure