NEVIS'22: A Stream of 100 Tasks Sampled from 30 Years of Computer Vision Research

Jörg Bornschein, Alexandre Galashov, Ross Hemsley, Amal Rannen-Triki, Yutian Chen, Arslan Chaudhry, Xu Owen He, Arthur Douillard, Massimo Caccia, Qixuang Feng, Jiajun Shen, Sylvestre-Alvise Rebuffi, Kitty Stacpoole, Diego de Las Casas, Will Hawkins, Angeliki Lazaridou, Yee Whye Teh, Andrei A. Rusu, Razvan Pascanu, Marc'Aurelio Ranzato

TL;DR

This paper introduces NEVIS'22, a novel benchmark designed to evaluate how well machine learning models can learn and adapt to a continuous stream of diverse visual classification tasks over time. It addresses the critical need for benchmarks that go beyond static datasets to better reflect real-world scenarios where models must continuously learn and improve, emphasizing both accuracy and computational efficiency.

What Problem Does This Paper Address?

The field of machine learning, particularly in areas like continual learning, meta-learning, and transfer learning, aims to build models that can adapt to new tasks efficiently and robustly. The ultimate goal is to create systems that learn continuously, becoming more adept over time by leveraging past knowledge. However, current benchmarks often fall short by being too small, homogeneous, or not reflecting the chronological evolution of research. This makes it difficult to assess how well models truly adapt to unseen tasks and improve their learning efficiency. NEVIS'22 tackles this by creating a large-scale, diverse, and chronologically ordered stream of tasks to provide a more realistic testbed for these "never-ending" learning systems.

Methods

The core contribution is the NEVIS'22 benchmark, constructed with four principles: reproducibility, simplicity (well-understood tasks), agnostic task selection (no bias towards specific approaches), and scale (intermediate, large enough to be challenging but allowing fast iteration). The benchmark is built by:

  1. Gathering papers from leading computer vision conferences over three decades.
  2. Sampling papers randomly from each year.
  3. Extracting classification tasks used for empirical validation.
  4. Filtering for publicly available datasets with permissive licenses.
  5. Removing duplicates within a 10-year window to avoid redundancy.

The paper also proposes a rigorous training and evaluation protocol that includes:

  • A meta-training stream (STr) for tuning meta-learner hyperparameters and a meta-test stream (STs) for final evaluation.
  • A strict rule: learners cannot use future task information to influence their behavior on current tasks.
  • Accounting for computational cost (cFLOPS) alongside accuracy, encouraging efficient methods.
  • Defining baselines that represent various transfer learning strategies (independent learning, fine-tuning, multi-tasking, pre-training).

Datasets

NEVIS'22 is a benchmark consisting of 106 image classification datasets totaling approximately 8 million images, sampled from computer vision research over the last 30 years. The datasets span a wide range of domains (OCR, texture, scene recognition, object recognition, medical, etc.) and dataset sizes, from small to very large.

The paper also uses a SHORT stream variant, created by randomly selecting two datasets per year from the original list, resulting in 24 tasks, for faster experimentation.

Key Insights and Ideas

  • Chronological ordering matters: The way tasks are presented significantly impacts performance, especially for large datasets like ImageNet. Placing them earlier can lead to better overall results.
  • Transfer learning is crucial, but not all transfer is equal: While methods that allow transfer learning generally outperform independent learners, the effectiveness varies greatly by domain. Some domains (like OCR) show minimal gains, while others (like object recognition) benefit substantially.
  • Pre-training is powerful, but adaptation is key: Pre-trained models offer a strong starting point, but they don't inherently accrue knowledge over time. Combining pre-training with sequential fine-tuning (PT+FT) can further improve performance by adapting general representations to specific tasks.
  • Task relatedness estimation is critical for fine-tuning: The performance of fine-tuning methods (like FT-d) is highly dependent on how well they can estimate the relatedness of previous tasks for initialization. FT-prev (using the most recent task) performs significantly worse than FT-d (using a more dynamic approach).
  • Current methods struggle with "minor" domains and continuous improvement: Even advanced methods fail to transfer well to certain domains (e.g., medical, OCR) and, importantly, none of the tested baselines showed improvement over time as they encountered more data and tasks. This highlights a significant gap in achieving true "never-ending" learning.

Key Results

  • Significant performance gaps: A gap of about 5% absolute error exists between the best and worst methods, with independent learning (Indep) being the baseline.
  • FT-d excels: The FT-d (fine-tuning with dynamic task selection) baseline, especially when combined with pre-training (PT-ext + FT-d), achieved the lowest average error rates.
  • Compute-efficiency trade-offs: Methods that leverage transfer learning generally offer better trade-offs between accuracy and compute. Multi-tasking (MT) methods, while potentially achieving lower error, often incur higher computational costs without a better Pareto front.
  • Domain-specific performance: PT-ext was the best method in the medical domain but the worst in OCR, demonstrating the importance of domain-specific adaptation.
  • Forward transfer is positive but limited: Fine-tuning strategies achieved positive forward transfer on average, with FT-s outperforming FT-prev, indicating the importance of accurate task relatedness estimation.

Limitations and Future Work

  • Dataset limitations: The benchmark relies on older datasets, some of which may have issues with demographic representation or consent. The authors plan to update the benchmark regularly to address this.
  • Computational cost: Running the full benchmark can be computationally intensive, though a "SHORT" stream is provided for researchers with limited resources.
  • No improvement over time: A key limitation is that none of the tested methods showed improvement in accuracy as they encountered more data and tasks over time. This suggests a need for more sophisticated mechanisms for knowledge accumulation and adaptation.
  • Future directions: The authors suggest exploring architectures that support variable resolution inputs, efficient inference, and learning algorithms that enable better transfer, model growth, and parameter sharing. They also envision extending NEVIS'22 to multi-task and multi-modal streams. The benchmark itself is intended to evolve, with plans to update the meta-test stream and incorporate newer tasks.

Summary updated

Abstract

A shared goal of several machine learning communities like continual learning, meta-learning and transfer learning, is to design algorithms and models that efficiently and robustly adapt to unseen tasks. An even more ambitious goal is to build models that never stop adapting, and that become increasingly more efficient through time by suitably transferring the accrued knowledge. Beyond the study of the actual learning algorithm and model architecture, there are several hurdles towards our quest to build such models, such as the choice of learning protocol, metric of success and data needed to validate research hypotheses. In this work, we introduce the Never-Ending VIsual-classification Stream (NEVIS'22), a benchmark consisting of a stream of over 100 visual classification tasks, sorted chronologically and extracted from papers sampled uniformly from computer vision proceedings spanning the last three decades. The resulting stream reflects what the research community thought was meaningful at any point in time, and it serves as an ideal test bed to assess how well models can adapt to new tasks, and do so better and more efficiently as time goes by. Despite being limited to classification, the resulting stream has a rich diversity of tasks from OCR, to texture analysis, scene recognition, and so forth. The diversity is also reflected in the wide range of dataset sizes, spanning over four orders of magnitude. Overall, NEVIS'22 poses an unprecedented challenge for current sequential learning approaches due to the scale and diversity of tasks, yet with a low entry barrier as it is limited to a single modality and well understood supervised learning problems. Moreover, we provide a reference implementation including strong baselines and an evaluation protocol to compare methods in terms of their trade-off between accuracy and compute.
BibTeX
@article{bornschein2023nevis,
  author = {Jörg Bornschein and Alexandre Galashov and Ross Hemsley and Amal Rannen-Triki and Yutian Chen and Arslan Chaudhry and Xu Owen He and Arthur Douillard and Massimo Caccia and Qixuang Feng and Jiajun Shen and Sylvestre-Alvise Rebuffi and Kitty Stacpoole and Diego de Las Casas and Will Hawkins and Angeliki Lazaridou and Yee Whye Teh and Andrei A. Rusu and Razvan Pascanu and Marc'Aurelio Ranzato},
  title = {{NEVIS'22: A Stream of 100 Tasks Sampled from 30 Years of Computer Vision Research}},
  journal = {J. Mach. Learn. Res.},
  year = {2023},
  url = {https://openreview.net/forum?id=QhxCKqf7Id},
}
Figure