Is Multi-Task Learning an Upper Bound for Continual Learning?

Zihao Wu, Huy Tran, Hamed Pirsiavash, Soheil Kolouri

TL;DR

This paper challenges the long-standing assumption that Multi-Task Learning (MTL) represents the absolute performance "ceiling" for Continual Learning (CL). By framing the learning of different image augmentations as sequential tasks in a self-supervised setting, the authors demonstrate that CL can actually outperform MTL by avoiding the "interference" and negative transfer that occurs when conflicting tasks are trained simultaneously.

What Problem Does This Paper Address?

In the world of Machine Learning, we usually compare two ways of learning multiple tasks:

  1. Multi-Task Learning (MTL): Training on all tasks at once (joint training).
  2. Continual Learning (CL): Training on tasks one after another (sequential training).

For years, the consensus has been that MTL is the "upper bound"—the best possible performance—because the model has access to all data simultaneously. CL is usually seen as a struggle to reach that peak while fighting "catastrophic forgetting" (losing old knowledge).

The authors ask: Is MTL actually the upper bound? They argue that when tasks are "adversarial" or conflicting, training on them all at once can cause interference that degrades performance. They suggest that learning tasks sequentially might actually allow a model to build a more robust representation by focusing on one invariance at a time.

Methods

The authors test their hypothesis using Self-Supervised Learning (SSL). In SSL, models learn by trying to make the representations of two different "views" of the same image (created via augmentations like cropping or color jittering) similar.

  • Tasks as Augmentations: Instead of different datasets, the authors define each "task" as a specific type of image augmentation (e.g., Task 1 is Cropping, Task 2 is Gaussian Noise).
  • Core Algorithm: They use Barlow Twins, a non-contrastive SSL method that focuses on redundancy reduction in the cross-correlation matrix of embeddings.
  • The CL Framework: To prevent forgetting during sequential training, they use CaSSLe (Continual Self-Supervised Learning). CaSSLe uses a distillation loss to ensure the current model can still predict the embeddings that the previous model would have generated.
  • The Comparison: They compare this sequential approach against a standard MTL baseline where the model is trained on all five augmentations simultaneously for an equivalent number of epochs.

Datasets

The experiments were conducted on three standard computer vision benchmarks:

  • MNIST: Handwritten digits.
  • CIFAR-10: 10 classes of common objects/animals.
  • CIFAR-100: 100 classes of common objects/animals.

For each dataset, they generated five different "curricula" (random orderings of the augmentation tasks) to ensure the results weren't just a fluke of task order.

Key Insights and Ideas

  • Interference is Real: In MTL, different tasks can act as "adversaries." For example, on CIFAR-10, training on "Cropping" alone yields 53.6% accuracy. When you add "Flipping" in an MTL setup, accuracy drops to 50.5%. This is negative transfer.
  • CL as a Buffer: The sequential nature of CL, combined with distillation, seems to act as a buffer. In the same Cropping/Flipping example, the CL approach only dropped to 52.6%, outperforming the "upper bound" MTL.
  • Invariance Accumulation: The paper suggests that CL allows the model to "accumulate" different types of spatial and color invariances one by one, rather than trying to resolve the conflicting gradients of all augmentations at once.

Key Results

The results consistently showed CL matching or beating MTL across various curricula:

  • CIFAR-10: In one curriculum, the final CL accuracy was 60.90%, while the MTL baseline was only 50.55%.
  • CIFAR-100: CL reached 33.77% compared to MTL's 25.76%.
  • Negative Transfer: The authors quantified "Average Negative Transfer." On CIFAR-10, MTL had a negative transfer of -3.14, while CL was significantly lower at -0.44. This proves that CL is much better at preserving performance when new, potentially conflicting tasks are introduced.

Limitations and Future Work

  • Scale: The study was limited to relatively small datasets (MNIST and CIFAR). The authors acknowledge that testing on large-scale datasets like ImageNet is a necessary next step.
  • Task Definition: The "tasks" here are specific augmentations within an SSL framework. It remains to be seen if CL can beat MTL in more traditional supervised settings (like learning different object classes) where tasks are less likely to be "naturally adversarial" in the same way.
  • Curriculum Sensitivity: While they tested five curricula, the order of tasks still matters for CL performance, and finding the optimal order remains an open challenge.

Summary updated

Abstract

Continual and multi-task learning are common machine learning approaches to learning from multiple tasks. The existing works in the literature often assume multi-task learning as a sensible performance upper bound for various continual learning algorithms. While this assumption is empirically verified for different continual learning benchmarks, it is not rigorously justified. Moreover, it is imaginable that when learning from multiple tasks, a small subset of these tasks could behave as adversarial tasks reducing the overall learning performance in a multi-task setting. In contrast, continual learning approaches can avoid the performance drop caused by such adversarial tasks to preserve their performance on the rest of the tasks, leading to better performance than a multi-task learner. This paper proposes a novel continual self-supervised learning setting, where each task corresponds to learning an invariant representation for a specific class of data augmentations. In this setting, we show that continual learning often beats multi-task learning on various benchmark datasets, including MNIST, CIFAR-10, and CIFAR-100.
BibTeX
@inproceedings{wu2023is,
  author = {Zihao Wu and Huy Tran and Hamed Pirsiavash and Soheil Kolouri},
  title = {{Is Multi-Task Learning an Upper Bound for Continual Learning?}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2023},
  doi = {10.1109/icassp49357.2023.10095984},
}
Figure