TL;DR
This paper introduces NEVIS'22, a novel benchmark designed to evaluate how well machine learning models can learn and adapt to a continuous stream of diverse visual classification tasks over time. It addresses the critical need for benchmarks that go beyond static datasets to better reflect real-world scenarios where models must continuously learn and improve, emphasizing both accuracy and computational efficiency.
What Problem Does This Paper Address?
The field of machine learning, particularly in areas like continual learning, meta-learning, and transfer learning, aims to build models that can adapt to new tasks efficiently and robustly. The ultimate goal is to create systems that learn continuously, becoming more adept over time by leveraging past knowledge. However, current benchmarks often fall short by being too small, homogeneous, or not reflecting the chronological evolution of research. This makes it difficult to assess how well models truly adapt to unseen tasks and improve their learning efficiency. NEVIS'22 tackles this by creating a large-scale, diverse, and chronologically ordered stream of tasks to provide a more realistic testbed for these "never-ending" learning systems.
Methods
The core contribution is the NEVIS'22 benchmark, constructed with four principles: reproducibility, simplicity (well-understood tasks), agnostic task selection (no bias towards specific approaches), and scale (intermediate, large enough to be challenging but allowing fast iteration). The benchmark is built by:
- Gathering papers from leading computer vision conferences over three decades.
- Sampling papers randomly from each year.
- Extracting classification tasks used for empirical validation.
- Filtering for publicly available datasets with permissive licenses.
- Removing duplicates within a 10-year window to avoid redundancy.
The paper also proposes a rigorous training and evaluation protocol that includes:
- A meta-training stream (STr) for tuning meta-learner hyperparameters and a meta-test stream (STs) for final evaluation.
- A strict rule: learners cannot use future task information to influence their behavior on current tasks.
- Accounting for computational cost (cFLOPS) alongside accuracy, encouraging efficient methods.
- Defining baselines that represent various transfer learning strategies (independent learning, fine-tuning, multi-tasking, pre-training).
Datasets
NEVIS'22 is a benchmark consisting of 106 image classification datasets totaling approximately 8 million images, sampled from computer vision research over the last 30 years. The datasets span a wide range of domains (OCR, texture, scene recognition, object recognition, medical, etc.) and dataset sizes, from small to very large.
The paper also uses a SHORT stream variant, created by randomly selecting two datasets per year from the original list, resulting in 24 tasks, for faster experimentation.
Key Insights and Ideas
- Chronological ordering matters: The way tasks are presented significantly impacts performance, especially for large datasets like ImageNet. Placing them earlier can lead to better overall results.
- Transfer learning is crucial, but not all transfer is equal: While methods that allow transfer learning generally outperform independent learners, the effectiveness varies greatly by domain. Some domains (like OCR) show minimal gains, while others (like object recognition) benefit substantially.
- Pre-training is powerful, but adaptation is key: Pre-trained models offer a strong starting point, but they don't inherently accrue knowledge over time. Combining pre-training with sequential fine-tuning (PT+FT) can further improve performance by adapting general representations to specific tasks.
- Task relatedness estimation is critical for fine-tuning: The performance of fine-tuning methods (like FT-d) is highly dependent on how well they can estimate the relatedness of previous tasks for initialization. FT-prev (using the most recent task) performs significantly worse than FT-d (using a more dynamic approach).
- Current methods struggle with "minor" domains and continuous improvement: Even advanced methods fail to transfer well to certain domains (e.g., medical, OCR) and, importantly, none of the tested baselines showed improvement over time as they encountered more data and tasks. This highlights a significant gap in achieving true "never-ending" learning.
Key Results
- Significant performance gaps: A gap of about 5% absolute error exists between the best and worst methods, with independent learning (Indep) being the baseline.
- FT-d excels: The FT-d (fine-tuning with dynamic task selection) baseline, especially when combined with pre-training (PT-ext + FT-d), achieved the lowest average error rates.
- Compute-efficiency trade-offs: Methods that leverage transfer learning generally offer better trade-offs between accuracy and compute. Multi-tasking (MT) methods, while potentially achieving lower error, often incur higher computational costs without a better Pareto front.
- Domain-specific performance: PT-ext was the best method in the medical domain but the worst in OCR, demonstrating the importance of domain-specific adaptation.
- Forward transfer is positive but limited: Fine-tuning strategies achieved positive forward transfer on average, with FT-s outperforming FT-prev, indicating the importance of accurate task relatedness estimation.
Limitations and Future Work
- Dataset limitations: The benchmark relies on older datasets, some of which may have issues with demographic representation or consent. The authors plan to update the benchmark regularly to address this.
- Computational cost: Running the full benchmark can be computationally intensive, though a "SHORT" stream is provided for researchers with limited resources.
- No improvement over time: A key limitation is that none of the tested methods showed improvement in accuracy as they encountered more data and tasks over time. This suggests a need for more sophisticated mechanisms for knowledge accumulation and adaptation.
- Future directions: The authors suggest exploring architectures that support variable resolution inputs, efficient inference, and learning algorithms that enable better transfer, model growth, and parameter sharing. They also envision extending NEVIS'22 to multi-task and multi-modal streams. The benchmark itself is intended to evolve, with plans to update the meta-test stream and incorporate newer tasks.
Summary updated