CrossFit: A Few-shot Learning Challenge for Cross-task Generalization in NLP

Qinyuan Ye, Bill Yuchen Lin, Xiang Ren

Abstract

CrossFit is a novel benchmark and challenge designed to evaluate the ability of NLP models to "learn to learn" across diverse tasks. By leveraging a "Gym" of 160 NLP tasks converted into a unified text-to-text format, the researchers demonstrate that models can significantly improve few-shot performance on entirely new task types. The study finds that simple multitask learning often rivals or outperforms complex meta-learning algorithms, highlighting the importance of task diversity in cross-task generalization.

Introduction

While few-shot learning research in NLP typically focuses on instance-level generalization—learning a specific task from a few examples—humans excel at task-level generalization. We use experience from prior, unrelated tasks to master new ones rapidly. CrossFit addresses the lack of standardized measurement for this "cross-task" ability by providing a formal problem setup, a massive repository of diverse tasks, and a rigorous evaluation protocol to test if models can leverage "seen" tasks to master "unseen" ones.

The CrossFit Framework

The authors propose a two-stage learning framework to facilitate cross-task transfer:

  1. Upstream Learning: The model is trained on a set of "seen" tasks. Approaches include Multi-task Learning (MTL), where task data is mixed, and various Meta-learning algorithms (e.g., MAML, Reptile) designed to optimize for rapid adaptation.
  2. Downstream Fine-tuning: The resulting model is fine-tuned on a small number of examples for an "unseen" task.
The CrossFit Challenge involves upstream learning on training tasks followed by downstream fine-tuning on test tasks using a unified text-to-text transformer.

To enable this across 160 disparate tasks, all inputs and outputs are converted into a unified text-to-text format. This allows a single backbone transformer, such as BART-Base, to process everything from classification to summarization.

The NLP Few-shot Gym

The core of this work is the NLP Few-shot Gym, a curated repository of 160 diverse NLP tasks from HuggingFace Datasets.

The NLP Few-shot Gym includes 160 tasks across categories like Classification, Question Answering, and Conditional Generation.

The Gym is characterized by:

  • Diversity: Tasks span sentiment analysis, NLI, multiple-choice QA, reading comprehension, summarization, dialogue, and niche areas like acronym identification.
  • Strategic Partitions: Eight specific "seen/unseen" splits test different generalization types. For instance, one split holds out all NLI tasks to evaluate if a model trained on other categories can adapt to NLI.
  • Standardization: Each task uses a consistent few-shot size (e.g., 16 examples per class).

Key Insights and Results

The CrossFit experiments yielded several surprising findings:

  • The Power of Diversity: "Well-rounded" training on a mix of task types was highly effective. Interestingly, training on completely different task types (e.g., non-classification tasks) often helped as much as training on similar ones.
  • MTL vs. Meta-Learning: Despite the theoretical advantages of meta-learning, simple MTL proved to be a remarkably strong baseline, often outperforming complex optimization loops. This suggests that for large transformers, exposure to task diversity is the primary driver of performance.
  • Widespread Gains: Upstream learning improved few-shot performance on over 51% of test tasks by more than 5% relative gain. On the Random partition, models saw a 35% average improvement over direct fine-tuning.
  • Catastrophic Forgetting: In some cases (about 12% of tasks), upstream learning actually hurt performance. Tasks like CoLA (linguistic acceptability) suffered, likely because the model "forgot" fundamental language modeling knowledge during the upstream stage.

Conclusion and Future Directions

CrossFit demonstrates that cross-task generalization is a viable path toward more robust few-shot learners. Future work will need to address "negative transfer" and explore automated ways to measure task similarity for better task selection. Additionally, integrating these generalization benefits with prompting techniques remains a promising frontier for NLP research.

Summary updated

Abstract

Humans can learn a new language task efficiently with only few examples, by leveraging their knowledge obtained when learning prior tasks. In this paper, we explore whether and how such cross-task generalization ability can be acquired, and further applied to build better few-shot learners across diverse NLP tasks. We introduce CrossFit, a problem setup for studying cross-task generalization ability, which standardizes seen/unseen task partitions, data access during different learning stages, and the evaluation protocols. To instantiate different seen/unseen task partitions in CrossFit and facilitate in-depth analysis, we present the NLP Few-shot Gym, a repository of 160 diverse few-shot NLP tasks created from open-access NLP datasets and converted to a unified text-to-text format. Our analysis reveals that the few-shot learning ability on unseen tasks can be improved via an upstream learning stage using a set of seen tasks. We also observe that the selection of upstream learning tasks can significantly influence few-shot performance on unseen tasks, asking further analysis on task similarity and transferability.
BibTeX
@inproceedings{ye2021crossfit,
  author = {Qinyuan Ye and Bill Yuchen Lin and Xiang Ren},
  title = {{CrossFit: A Few-shot Learning Challenge for Cross-task Generalization in NLP}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2021},
  doi = {10.18653/v1/2021.emnlp-main.572},
}
Figure