Towards Redundancy-Free Sub-networks in Continual Learning

Cheng Chen, Jingkuan Song, LianLi Gao, Heng Tao Shen

TL;DR

The Information Bottleneck Masked sub-network (IBM) framework tackles catastrophic forgetting by using Information Bottleneck theory to identify and freeze only the most essential weights for each task. By eliminating redundancy, IBM achieves state-of-the-art accuracy while reducing sub-network parameters by 70% and slashing training time by 80% compared to previous methods.

What Problem Does This Paper Address?

Deep neural networks are notoriously bad at multitasking over time. When trained on a sequence of tasks, they suffer from Catastrophic Forgetting (CF)—the tendency to abruptly lose information from previous tasks as they learn new ones.

One of the most effective ways to stop this is parameter isolation: finding a specific "sub-network" within the larger model for each task and freezing those weights. However, current methods usually pick which weights to keep based on their magnitude (i.e., how large the weight value is). The authors argue that magnitude is a poor proxy for importance. This leads to "bloated" sub-networks that contain redundant, unimportant weights, which quickly exhausts the total capacity of the model and slows down training.

Methods

The authors propose IBM (Information Bottleneck Masked sub-network), which shifts the focus from weight magnitude to information theory.

  1. Information Bottleneck (IB) Objective: Instead of just looking at weight size, IBM uses the IB principle to compress the network. It optimizes a trade-off: maximize the information the hidden layers have about the target output while minimizing the redundant information they carry from the previous layer.
  2. Variational Masking: To make this mathematically tractable, they use variational inference to learn a distribution over the weights. This process naturally pushes "useless" information into expendable parameters and aggregates "valuable" information into a small set of essential weights.
  3. Feature Decomposing (FD) Module: Not all layers are created equal. Lower layers handle basic visual features, while higher layers handle complex semantics. IBM uses Singular Value Decomposition (SVD) to analyze hidden representations and automatically set the optimal compression ratio for each layer, rather than relying on manual tuning.
  4. Freeze and Re-init: Once a task is learned, the essential sub-network is frozen to prevent forgetting. The "expendable" weights are then re-initialized for the next task, which helps the model escape local optima and facilitates knowledge transfer.
Figure 2. An overview of the proposed IBM. After training on Task T, a binary mask is constructed based on choosing a subset of variational parameters. The mask and these parameters are copied and saved in a memory pool for inference. Then, before training on T+1, we maintain the chosen parameters and re-initialize the rest parameters to facilitate knowledge transfer. Finally, when training on Task T+1, indicated by the masks learned before, the corresponding weights of the network are frozen to solve catastrophic forgetting.

Datasets

The framework was evaluated on standard continual learning benchmarks:

  • CIFAR-100: Split into 10 sequential tasks.
  • TinyImageNet: A variant of ImageNet with 200 classes, split into 10 tasks.
  • MiniImageNet: 100 classes split into 10 tasks.
  • Extended Sequences: To test model capacity, they also ran 20-task evaluations on these datasets.

Key Insights and Ideas

  • Redundancy is the Enemy of Capacity: By using IB to prune redundancy, the model leaves more "room" for future tasks. This allows IBM to handle longer task sequences without running out of parameters.
  • Magnitude $\neq$ Importance: The visualization in Figure 3 shows that IBM selects far fewer weights in higher layers compared to magnitude-based methods like WSN, proving that those layers were previously full of redundant information.
  • Automated Layer Ratios: The Feature Decomposing module proves that dynamic, layer-wise compression is superior to a "one-size-fits-all" sparsity target across the whole network.
Figure 3. The visualization of masked weight numbers within each layer of the first sub-network constructed by WSN and IBM on CIFAR-100 with Resnet-18. The significant reduction of weight numbers substantiates the conjecture of employing the information bottleneck for reducing redundancy within sub-networks.

Key Results

IBM sets a new bar for efficiency and performance in parameter-isolation methods:

  • Efficiency: Compared to the previous state-of-the-art (WSN), IBM achieves a 70% reduction in parameters per sub-network and an 80% reduction in training time.
  • Accuracy: On CIFAR-100 with a ResNet-18 backbone, IBM achieved 88.15% average accuracy, surpassing WSN's 86.47%.
  • Forget-Free: Because it freezes sub-networks, it maintains 0% backward transfer (meaning it never forgets a task once learned).
  • Knowledge Transfer: IBM showed positive "Forward Transfer," meaning learning earlier tasks actually helped it learn later tasks faster and better than training from scratch.
Figure 1. The total number of parameters within sub-networks and average training time for each task of WSN [15] and our IBM on CIFAR-100 with Resnet-18. Significantly, our method surpasses WSN by 1.68% with a 70% reduction in the number of parameters and an 80% decrease in training time.

Limitations and Future Work

The authors note that as tasks become significantly more difficult, the extreme compactness of the sub-networks could potentially lead to "parameter deficiency," where the sub-network is too small to capture the task's complexity. While their current knowledge transfer mechanism mitigates this, future work could explore adaptive sub-network expansion for high-complexity tasks. Additionally, while the method works well on ResNet and AlexNet, testing on Transformer-based architectures remains an open area for exploration.

Summary updated

Abstract

Catastrophic Forgetting (CF) is a prominent issue in continual learning. Parameter isolation addresses this challenge by masking a sub-network for each task to mitigate interference with old tasks. However, these sub-networks are constructed relying on weight magnitude, which does not necessarily correspond to the importance of weights, resulting in maintaining unimportant weights and constructing redundant sub-networks. To overcome this limitation, inspired by information bottleneck, which removes redundancy between adjacent network layers, we propose \textbf{\underline{I}nformation \underline{B}ottleneck \underline{M}asked sub-network (IBM)} to eliminate redundancy within sub-networks. Specifically, IBM accumulates valuable information into essential weights to construct redundancy-free sub-networks, not only effectively mitigating CF by freezing the sub-networks but also facilitating new tasks training through the transfer of valuable knowledge. Additionally, IBM decomposes hidden representations to automate the construction process and make it flexible. Extensive experiments demonstrate that IBM consistently outperforms state-of-the-art methods. Notably, IBM surpasses the state-of-the-art parameter isolation method with a 70\% reduction in the number of parameters within sub-networks and an 80\% decrease in training time.
BibTeX
@article{chen2023towards,
  author = {Cheng Chen and Jingkuan Song and LianLi Gao and Heng Tao Shen},
  title = {{Towards Redundancy-Free Sub-networks in Continual Learning}},
  journal = {Pattern Recognition},
  year = {2023},
}
Figure