Adaptive Posterior Learning: few-shot learning with a surprise-based memory module

Tiago Ramalho, Marta GarneloICLR 2019

TL;DR

Adaptive Posterior Learning (APL) is a meta-learning algorithm that mimics Bayesian posterior updating by storing only the most "surprising" observations in an external memory module. By combining this efficient storage with a relational decoder, APL achieves state-of-the-art few-shot classification results and demonstrates a unique ability to perform deductive reasoning, sometimes succeeding with fewer than one example per class.

The Challenge of Few-Shot Learning

Traditional deep learning models are notoriously "data hungry," struggling to generalize when faced with new distributions or tasks after their initial training. Few-shot learning aims to bridge this gap, allowing models to learn from just a handful of examples.

However, existing few-shot methods face three major hurdles:

  1. Scalability: Many models struggle to handle thousands of potential labels.
  2. Efficiency: Storing every single observation is redundant and computationally expensive.
  3. Reasoning: Most models perform simple interpolation (finding the "closest" match via nearest neighbors) rather than complex logical deduction.

The authors introduce APL to tackle these issues by creating a system that knows what to remember and how to reason over those memories.

Methods: The APL Architecture

APL consists of four primary components designed to efficiently encode, store, and reason over data.

Overview of the Adaptive Posterior Learning (APL) architecture.
  • Encoder: A standard Convolutional Neural Network (CNN) that turns raw data (like images) into compact vector representations (embeddings).
  • External Memory Store: A database that holds pairs of embeddings and their true labels.
  • Relational Decoder: Unlike simple k-Nearest Neighbor lookups, APL uses a decoder (the best being a Relational Self-Attention module) that compares the current query against multiple recalled memories simultaneously. This allows the model to perform "all-to-all" attention and complex reasoning.
  • Memory Controller (The "Surprise" Mechanism): This is the core innovation. Instead of storing everything, the controller calculates "surprise"—defined as the negative log-likelihood of the correct label.
Sequential memory writing process in APL.

As shown in the sequential process above, if the model is already confident and correct about an observation, it discards the data. If the surprise exceeds a specific threshold ($\sigma$), the data is written to memory. This avoids the need for expensive backpropagation through the memory module and keeps the memory footprint small.

Key Insights and Ideas

  • Surprise as a Filter: By only storing surprising data, APL becomes highly memory-efficient. Interestingly, the authors found that surprise becomes "bimodal" during training: a model either knows exactly what an object is, or it is completely baffled. This makes the thresholding mechanism very robust.
  • Deductive Reasoning: Because the relational decoder looks at the relationships between memories, it can perform logical elimination—essentially "learning to learn" the underlying rules of a task rather than just matching patterns.
  • No Backprop Through Memory: By using a hard-coded surprise signal to trigger writes, the authors bypass the "credit assignment" problem that plagues differentiable memories (like MANNs). This makes the system faster and easier to train.

Datasets and Key Results

The researchers evaluated APL across several benchmarks, proving its versatility and power:

  • Omniglot: On this "transpose" of MNIST (featuring 1,623 different handwritten characters), APL achieved 99.9% accuracy on 5-way classification and 97.2% on 20-way classification. It matched or exceeded state-of-the-art models like MAML and SNAIL while using a significantly smaller memory footprint. Furthermore, APL successfully scaled to 1000-way classification, maintaining high accuracy where many other few-shot models falter.
  • ImageNet: To test large-scale capabilities, APL was evaluated on ImageNet, reaching 86.7% top-1 accuracy in a 20-way challenge after just 50 iterations.
  • Number Analogy Task: A novel meta-learning task introduced in this paper requires the model to solve equations like $X + S = Y$ (where $X$ is an MNIST digit and $S$ is an unknown symbol). APL reached 97.6% accuracy, demonstrating the ability to generalize using fewer than one example per possible class combination through deductive logic.

Limitations and Future Work

While APL is a significant step forward, the authors note a few areas for future exploration:

  • Fixed Encoders: For the ImageNet experiments, the authors used a pre-trained Inception-ResNet-v2 encoder rather than training it end-to-end. End-to-end training would likely yield even better results.
  • Long-term Reuse: Currently, the memory is cleared at the start of each "episode." Exploring how the system could store and retrieve information for reuse across different episodes (long-term life-long learning) is a promising next step.
  • Threshold Tuning: While the surprise threshold is robust, finding the absolute optimal $\sigma$ for a specific task still requires some empirical testing.

Summary updated

BibTeX
@inproceedings{ramalho2019adaptive,
  author = {Tiago Ramalho and Marta Garnelo},
  title = {{Adaptive Posterior Learning: few-shot learning with a surprise-based memory module}},
  booktitle = {International Conference on Learning Representations},
  year = {2019},
}
Figure