NeurIPS’22 Cross-Domain MetaDL competition: Design and baseline results

Dustin Carrión-Ojeda, Hong Chen, Adrian El Baz, Sergio Escalera, Chaoyu Guan, Isabelle M Guyon, Ihsan Ullah, Xin Wang, Wenwu Zhu

TL;DR

This paper introduces the NeurIPS'22 Cross-Domain MetaDL competition, a new challenge that pushes meta-learning beyond traditional within-domain few-shot learning. By introducing "any-way, any-shot" tasks across diverse domains, the competition aims to foster more robust and generalizable few-shot learning models, with a new benchmark dataset, Meta-Album, to support this goal.

What Problem Does This Paper Address?

Traditional few-shot learning (FSL) and meta-learning challenges often focus on "within-domain" scenarios. This means models are trained and tested on data from the same general domain (e.g., classifying different types of insects after being trained on insect images). While this has led to impressive accuracy gains, it doesn't reflect real-world scenarios where we might need to classify rare species of plants or medical images with very limited data, and the available training data comes from completely different domains. Models trained within a domain often struggle to generalize to unrelated domains. This paper addresses this gap by proposing a "cross-domain" meta-learning challenge, where models must learn to adapt to new tasks from unseen domains with varying numbers of classes ("ways") and examples per class ("shots"). This is crucial for building more practical and broadly applicable AI systems.

Methods

The core of this work is the design of the NeurIPS'22 Cross-Domain MetaDL competition. Key methodological aspects include:

  • Cross-Domain Meta-Learning: Unlike previous challenges, this competition explicitly requires models to generalize across domains. This means meta-training data and meta-testing data come from different, unrelated domains.
  • "Any-Way, Any-Shot" Setting: The challenge moves beyond fixed N-way k-shot tasks. Participants must handle tasks with a variable number of classes (2-20 ways) and a variable number of training examples per class (1-20 shots). This makes the problem significantly more challenging and realistic.
  • Meta-Album Dataset: A new meta-dataset, Meta-Album, was created for this competition. It comprises 40 image classification datasets from 10 diverse domains (e.g., healthcare, ecology, biology, manufacturing), specifically chosen for their societal impact. This dataset is partitioned into sets for meta-training, meta-validation, and meta-testing.
  • Competition Protocol: The competition involves three stages: meta-training, meta-validation (optional), and meta-testing. Submissions are code-based and evaluated on a blind test set.
  • Baseline Methods: The paper evaluates six baseline methods, including simple "train-from-scratch" and "fine-tuning" approaches, as well as established meta-learning algorithms like Matching Networks, Prototypical Networks, FO-MAML, and MetaDelta++. These baselines are tested with both randomly initialized and pre-trained backbones to understand the impact of initialization.

Datasets

The primary dataset introduced and used for this competition is Meta-Album.

  • Meta-Album: This is a collection of 40 image classification datasets spanning 10 diverse domains. For the competition, 30 of these datasets were used, partitioned into three sets (Set-0, Set-1, Set-2) for different phases of the competition. The preprocessing pipeline standardizes images to 128x128x3 pixels.
  • Comparison to Meta-Dataset: The authors note that while Meta-Dataset is a popular benchmark for cross-domain meta-learning, it was not used here because it is too well-known by the community, and its datasets have a large variance in class and example counts, which could introduce bias.

Key Insights and Ideas

  • Cross-domain generalization is significantly harder: The results clearly show that performance drops substantially when moving from within-domain to cross-domain settings, highlighting the challenge's difficulty.
  • Pre-trained backbones are a major advantage: Across most methods, using a backbone pre-trained on ImageNet leads to significantly better performance compared to training from scratch. This suggests that leveraging large-scale pre-training is a powerful shortcut, potentially making meta-learning itself less critical for image classification in some scenarios.
  • Meta-learning is still valuable when pre-training is not an option: For scenarios where pre-trained backbones are unavailable (e.g., specialized domains or privacy concerns), meta-learning methods like Prototypical Networks show strong performance, especially when random initialization is used.
  • The "any-way, any-shot" setting introduces complexity: The performance of methods degrades with an increasing number of ways and generally improves with more shots, but the cross-domain nature adds a layer of difficulty that is not simply additive.
  • Dataset difficulty varies: The competition datasets exhibit a range of intrinsic and modeling difficulties, which is desirable for a challenge to effectively differentiate methods.

Key Results

  • Performance drop in cross-domain: For example, the best performing baseline (MetaDelta++) achieved 75.8% accuracy in the within-domain setting with a pre-trained backbone, but this dropped to 62.3% in the cross-domain setting.
  • Pre-training benefit: With a pre-trained backbone, MetaDelta++ achieved 75.8% accuracy in the within-domain setting, compared to 23.6% when trained from scratch.
  • Influence of ways and shots: In the cross-domain setting with pre-trained backbones, performance generally degrades as the number of "ways" increases and improves as the number of "shots" increases. The most significant gains from increasing shots are observed up to 5 shots.
  • Baseline performance: Among the baselines, MetaDelta++ (winner of the previous within-domain challenge) performed best when using pre-trained backbones. When random initialization was required, Prototypical Networks often showed the best performance.

Limitations and Future Work

  • Meta-training data size: The authors suggest that the meta-training dataset might not be large enough, or the meta-training time insufficient, which could explain the significant benefit of pre-trained backbones.
  • Domain-specific meta-learning: Some datasets showed a significant performance drop in the cross-domain setting compared to within-domain, suggesting that domain-specific meta-learning might be more favorable for certain types of data.
  • Future Work: The authors plan to organize a "domain independent" sequel to the challenge, where meta-testing will involve datasets from entirely new domains not seen during meta-training. This would further push the boundaries of generalization.

Summary updated

Abstract

We present the design and baseline results for a new challenge in the ChaLearn meta-learning series, accepted at NeurIPS'22, focusing on "cross-domain" meta-learning. Meta-learning aims to leverage experience gained from previous tasks to solve new tasks efficiently (i.e., with better performance, little training data, and/or modest computational resources). While previous challenges in the series focused on within-domain few-shot learning problems, with the aim of learning efficiently N-way k-shot tasks (i.e., N class classification problems with k training examples), this competition challenges the participants to solve "any-way" and "any-shot" problems drawn from various domains (healthcare, ecology, biology, manufacturing, and others), chosen for their humanitarian and societal impact. To that end, we created Meta-Album, a meta-dataset of 40 image classification datasets from 10 domains, from which we carve out tasks with any number of "ways" (within the range 2-20) and any number of "shots" (within the range 1-20). The competition is with code submission, fully blind-tested on the CodaLab challenge platform. The code of the winners will be open-sourced, enabling the deployment of automated machine learning solutions for few-shot image classification across several domains.
BibTeX
@inproceedings{carrionojeda2022neurips,
  author = {Dustin Carrión-Ojeda and Hong Chen and Adrian EL BAZ and Sergio Escalera and Chaoyu Guan and Isabelle M Guyon and Ihsan Ullah and Xin Wang and Wenwu Zhu},
  title = {{NeurIPS’22 Cross-Domain MetaDL competition: Design and baseline results}},
  booktitle = {Meta-Knowledge Transfer @ ECML/PKDD},
  year = {2022},
  url = {https://openreview.net/forum?id=fiOpqxnRhyZn},
}
Figure