TL;DR
The Information Bottleneck Masked sub-network (IBM) framework tackles catastrophic forgetting by using Information Bottleneck theory to identify and freeze only the most essential weights for each task. By eliminating redundancy, IBM achieves state-of-the-art accuracy while reducing sub-network parameters by 70% and slashing training time by 80% compared to previous methods.
What Problem Does This Paper Address?
Deep neural networks are notoriously bad at multitasking over time. When trained on a sequence of tasks, they suffer from Catastrophic Forgetting (CF)—the tendency to abruptly lose information from previous tasks as they learn new ones.
One of the most effective ways to stop this is parameter isolation: finding a specific "sub-network" within the larger model for each task and freezing those weights. However, current methods usually pick which weights to keep based on their magnitude (i.e., how large the weight value is). The authors argue that magnitude is a poor proxy for importance. This leads to "bloated" sub-networks that contain redundant, unimportant weights, which quickly exhausts the total capacity of the model and slows down training.
Methods
The authors propose IBM (Information Bottleneck Masked sub-network), which shifts the focus from weight magnitude to information theory.
- Information Bottleneck (IB) Objective: Instead of just looking at weight size, IBM uses the IB principle to compress the network. It optimizes a trade-off: maximize the information the hidden layers have about the target output while minimizing the redundant information they carry from the previous layer.
- Variational Masking: To make this mathematically tractable, they use variational inference to learn a distribution over the weights. This process naturally pushes "useless" information into expendable parameters and aggregates "valuable" information into a small set of essential weights.
- Feature Decomposing (FD) Module: Not all layers are created equal. Lower layers handle basic visual features, while higher layers handle complex semantics. IBM uses Singular Value Decomposition (SVD) to analyze hidden representations and automatically set the optimal compression ratio for each layer, rather than relying on manual tuning.
- Freeze and Re-init: Once a task is learned, the essential sub-network is frozen to prevent forgetting. The "expendable" weights are then re-initialized for the next task, which helps the model escape local optima and facilitates knowledge transfer.
Datasets
The framework was evaluated on standard continual learning benchmarks:
- CIFAR-100: Split into 10 sequential tasks.
- TinyImageNet: A variant of ImageNet with 200 classes, split into 10 tasks.
- MiniImageNet: 100 classes split into 10 tasks.
- Extended Sequences: To test model capacity, they also ran 20-task evaluations on these datasets.
Key Insights and Ideas
- Redundancy is the Enemy of Capacity: By using IB to prune redundancy, the model leaves more "room" for future tasks. This allows IBM to handle longer task sequences without running out of parameters.
- Magnitude $\neq$ Importance: The visualization in Figure 3 shows that IBM selects far fewer weights in higher layers compared to magnitude-based methods like WSN, proving that those layers were previously full of redundant information.
- Automated Layer Ratios: The Feature Decomposing module proves that dynamic, layer-wise compression is superior to a "one-size-fits-all" sparsity target across the whole network.
Key Results
IBM sets a new bar for efficiency and performance in parameter-isolation methods:
- Efficiency: Compared to the previous state-of-the-art (WSN), IBM achieves a 70% reduction in parameters per sub-network and an 80% reduction in training time.
- Accuracy: On CIFAR-100 with a ResNet-18 backbone, IBM achieved 88.15% average accuracy, surpassing WSN's 86.47%.
- Forget-Free: Because it freezes sub-networks, it maintains 0% backward transfer (meaning it never forgets a task once learned).
- Knowledge Transfer: IBM showed positive "Forward Transfer," meaning learning earlier tasks actually helped it learn later tasks faster and better than training from scratch.
Limitations and Future Work
The authors note that as tasks become significantly more difficult, the extreme compactness of the sub-networks could potentially lead to "parameter deficiency," where the sub-network is too small to capture the task's complexity. While their current knowledge transfer mechanism mitigates this, future work could explore adaptive sub-network expansion for high-complexity tasks. Additionally, while the method works well on ResNet and AlexNet, testing on Transformer-based architectures remains an open area for exploration.
Summary updated