Overview
Incremental Moment Matching (IMM) tackles catastrophic forgetting by merging the "knowledge" of multiple neural networks trained on different tasks into a single model. By treating network parameters as probability distributions and matching their moments (mean and variance), IMM achieves state-of-the-art performance on continual learning benchmarks without the memory overhead of ensembles.
The Challenge of Catastrophic Forgetting
Neural networks are notoriously bad at multitasking sequentially. When a model trained on Task A is subsequently trained on Task B, it typically "overwrites" the weights essential for Task A, leading to a total loss of previous knowledge—a phenomenon known as catastrophic forgetting.
Existing solutions often involve keeping an ensemble of models (which is computationally expensive) or using heavy regularization (like Elastic Weight Consolidation) that can be brittle if the tasks are very different. The authors of this paper seek a way to "merge" the parameters of models trained on sequential tasks so that the final model performs well on everything it has seen, using the framework of Bayesian Neural Networks.
Methods
The core idea is Incremental Moment Matching (IMM). The authors assume that the parameters of a network trained on a specific task follow a Gaussian posterior distribution. To solve multiple tasks, they attempt to approximate a mixture of these Gaussians (one per task) with a single Gaussian distribution.
They propose two primary merging strategies:
- mean-IMM: A simple approach that averages the parameters (means) of the models. This is mathematically equivalent to minimizing the average KL-divergence between the task-specific distributions and the merged one.
- mode-IMM: A more sophisticated approach using a Laplacian approximation. It uses the Fisher Information Matrix (which represents the "importance" or precision of each weight) to find the mode (peak) of the combined distribution.
To make this merging successful, the authors argue that the "search space" of the models must be smooth and convex-like. They introduce three transfer techniques to ensure this:
- Weight-transfer: Initializing the new task's model with the weights of the old task's model.
- L2-transfer: A regularization term that keeps new weights close to old weights.
- Drop-transfer: A novel variant of dropout. Instead of zeroing out neurons during training, "dropped" neurons are set to their values from the previous task's model.
Datasets
The authors evaluated IMM across four diverse scenarios:
- Disjoint MNIST: The dataset is split (e.g., digits 0-4 then 5-9) to test how the model handles new classes.
- Shuffled MNIST: Input pixels are randomly permuted for each task, testing the model's ability to handle entirely different input distributions.
- ImageNet to CUB: A transfer learning task moving from general ImageNet classification to fine-grained bird classification (Caltech-UCSD Birds-200-2011).
- Lifelog Dataset: Real-world egocentric video data from wearable sensors, where the model must categorize locations and activities that change over time.
Key Insights and Ideas
- The "Smooth Path" Hypothesis: The most critical insight is that merging only works if there is a smooth, low-loss path between the solutions for Task A and Task B.
The authors show that weight-transfer is the essential ingredient that creates this path, effectively "aligning" the models in parameter space so they can be averaged meaningfully.
- Drop-transfer as an Anchor: Traditional dropout encourages redundancy. Drop-transfer goes further for continual learning; by reverting to old values instead of zero, it forces the network to find solutions for the new task that are fundamentally compatible with the old task's internal representations.
- Uncertainty Matters: Mode-IMM performs better than mean-IMM because it respects the "certainty" of weights. If Task A was very sensitive to a specific weight but Task B wasn't, mode-IMM will lean toward the Task A value for that specific parameter.
Key Results
- Superior Accuracy: On the difficult Disjoint MNIST task, IMM (with L2 and drop-transfer) achieved over 94% accuracy, significantly outperforming standard SGD (~71%) and even Elastic Weight Consolidation (~52% in their setup).
- Efficiency: In the Lifelog experiment, IMM achieved results competitive with "Dual-Memory" architectures (which use complex ensembles) while using only a single network for inference.
- Robustness: The authors demonstrated that while mean-IMM is sensitive to the mixing ratio, mode-IMM is much more robust, maintaining high accuracy across a wider range of balancing hyper-parameters.
Limitations and Future Work
- Diagonal Assumption: To keep computation tractable, the authors assume a diagonal covariance matrix (meaning they ignore correlations between different weights). They acknowledge this is a "naive" assumption that might limit performance on highly complex tasks.
- Task Heterogeneity: When tasks have different output layers (e.g., ImageNet vs. CUB), the Fisher Information for the final layer is zero for the "other" task. The authors had to implement a specific re-scaling heuristic to handle this.
- Future Directions: The authors suggest exploring more complex priors, such as Matrix Gaussian distributions, to capture the correlations between neurons that the current diagonal Fisher approximation misses.
Summary updated