Integral Continual Learning Along the Tangent Vector Field of Tasks

Tian Yu Liu, Aditya Golatkar, Stefano Soatto, Alessandro Achille

Abstract

We propose a lightweight continual learning method which incorporates information from specialized datasets incrementally, by integrating it along the vector field of "generalist" models. The tangent plane to the specialist model acts as a generalist guide and avoids the kind of over-fitting that leads to catastrophic forgetting, while exploiting the convexity of the optimization landscape in the tangent plane. It maintains a small fixed-size memory buffer, as low as 0.4% of the source datasets, which is updated by simple resampling. Our method achieves state-of-the-art across various buffer sizes for different datasets. Specifically, in the class-incremental setting we outperform the existing methods that do not require distillation by an average of 18.77% and 28.48%, for Seq-CIFAR-10 and Seq-TinyImageNet respectively. Our method can easily be combined with existing replay-based continual learning methods. When memory buffer constraints are relaxed to allow storage of metadata such as logits, we attain state-of-the-art accuracy with an error reduction of 17.84% towards the paragon performance on Seq-CIFAR-10.

Introduction

In the field of machine learning, achieving "continual learning" remains a significant challenge. This is the ability for a model to learn new tasks sequentially without overwriting previously acquired knowledge—a phenomenon known as catastrophic forgetting. Standard deep neural networks typically fail here; as they optimize for a new Task B, the weights optimized for Task A are lost.

Current mitigation strategies often rely on large replay buffers or massive foundation models, which are computationally expensive and memory-intensive. The researchers behind Integral Tangent Learning (ITL) propose a geometric solution that maintains a "generalist" model's performance while specializing in new tasks, using a remarkably small memory footprint.

The ITL Framework: Geometry and Optimization

ITL conceptualizes the space of model parameters as a geometric manifold. Instead of directly updating the model weights in a non-convex space where forgetting is likely, ITL operates in the local, linear approximation of the model: the tangent plane.

Geometric representation of Integral Tangent Learning on the model manifold M.

The learning process for each new task follows a structured cycle:

  1. Specialist Training: The model is first fine-tuned on the new task data to become a "specialist."
  2. Linearization: The model is linearized around this specialist point, creating a tangent plane ($T_pM$).
  3. Tangent Learning: A "tangent generalist" is trained within this linear space using a tiny buffer of data from previous tasks. Because the tangent space is linear, the optimization is convex, ensuring stability and avoiding poor local minima.
  4. Projected Distillation: The tangent generalist is projected back onto the original model manifold through distillation, updating the actual network weights.
  5. The Half-Pipe Trick: To navigate flat regions of the loss landscape, the researchers employ a "half-pipe jump," perturbing the final layer to ensure stochastic gradient descent remains effective.

Experimental Results

The ITL method was rigorously tested on standard sequential benchmarks, including Seq-CIFAR-10 (5 tasks) and Seq-TinyImageNet (10 tasks). The experiments focused on "lightweight" scenarios with extremely limited buffer sizes, sometimes as low as 0.4% of the total training data.

Comparison of task accuracy over time between Full Fine-tuning and Integral Tangent Learning.

As shown in the heatmaps above, ITL significantly mitigates forgetting compared to standard full fine-tuning. While fine-tuning shows a rapid decline in accuracy for earlier tasks (top rows) as more tasks are added (moving right), ITL maintains much higher performance across the entire task history.

Key performance metrics include:

  • Accuracy Gains: ITL outperformed non-distillation methods by an average of 18.77% on Seq-CIFAR-10 and 28.48% on Seq-TinyImageNet.
  • Metadata Efficiency: A variant, ITL-L, which stores logits in the buffer, achieved a 17.84% error reduction compared to previous state-of-the-art methods.
  • Computational Overhead: The additional geometric steps are efficient, adding only about 10% to the total training time on Seq-CIFAR-10.

Conclusion and Future Directions

Integral Tangent Learning demonstrates that geometric intuition can solve fundamental problems in deep learning optimization. By leveraging the convexity of the tangent space, ITL provides a robust, lightweight framework for continual learning.

Future work may address current limitations, such as handling conflicting tasks where knowledge from one task might interfere with another. Additionally, scaling these geometric principles to massive Transformer architectures and diverse data modalities (text, audio) remains an open and promising area of research.

Summary updated

Abstract

We propose a lightweight continual learning method which incorporates information from specialized datasets incrementally, by integrating it along the vector field of "generalist" models. The tangent plane to the specialist model acts as a generalist guide and avoids the kind of over-fitting that leads to catastrophic forgetting, while exploiting the convexity of the optimization landscape in the tangent plane. It maintains a small fixed-size memory buffer, as low as 0.4% of the source datasets, which is updated by simple resampling. Our method achieves state-of-the-art across various buffer sizes for different datasets. Specifically, in the class-incremental setting we outperform the existing methods that do not require distillation by an average of 18.77% and 28.48%, for Seq-CIFAR-10 and Seq-TinyImageNet respectively. Our method can easily be combined with existing replay-based continual learning methods. When memory buffer constraints are relaxed to allow storage of metadata such as logits, we attain state-of-the-art accuracy with an error reduction of 17.84% towards the paragon performance on Seq-CIFAR-10.
BibTeX
@misc{liu2022integral,
  author = {Tian Yu Liu and Aditya Golatkar and Stefano Soatto and Alessandro Achille},
  title = {{Integral Continual Learning Along the Tangent Vector Field of Tasks}},
  year = {2022},
  eprint = {2211.13108},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2211.13108},
}
Figure