Abstract
CrossFit is a novel benchmark and challenge designed to evaluate the ability of NLP models to "learn to learn" across diverse tasks. By leveraging a "Gym" of 160 NLP tasks converted into a unified text-to-text format, the researchers demonstrate that models can significantly improve few-shot performance on entirely new task types. The study finds that simple multitask learning often rivals or outperforms complex meta-learning algorithms, highlighting the importance of task diversity in cross-task generalization.
Introduction
While few-shot learning research in NLP typically focuses on instance-level generalization—learning a specific task from a few examples—humans excel at task-level generalization. We use experience from prior, unrelated tasks to master new ones rapidly. CrossFit addresses the lack of standardized measurement for this "cross-task" ability by providing a formal problem setup, a massive repository of diverse tasks, and a rigorous evaluation protocol to test if models can leverage "seen" tasks to master "unseen" ones.
The CrossFit Framework
The authors propose a two-stage learning framework to facilitate cross-task transfer:
- Upstream Learning: The model is trained on a set of "seen" tasks. Approaches include Multi-task Learning (MTL), where task data is mixed, and various Meta-learning algorithms (e.g., MAML, Reptile) designed to optimize for rapid adaptation.
- Downstream Fine-tuning: The resulting model is fine-tuned on a small number of examples for an "unseen" task.
To enable this across 160 disparate tasks, all inputs and outputs are converted into a unified text-to-text format. This allows a single backbone transformer, such as BART-Base, to process everything from classification to summarization.
The NLP Few-shot Gym
The core of this work is the NLP Few-shot Gym, a curated repository of 160 diverse NLP tasks from HuggingFace Datasets.
The Gym is characterized by:
- Diversity: Tasks span sentiment analysis, NLI, multiple-choice QA, reading comprehension, summarization, dialogue, and niche areas like acronym identification.
- Strategic Partitions: Eight specific "seen/unseen" splits test different generalization types. For instance, one split holds out all NLI tasks to evaluate if a model trained on other categories can adapt to NLI.
- Standardization: Each task uses a consistent few-shot size (e.g., 16 examples per class).
Key Insights and Results
The CrossFit experiments yielded several surprising findings:
- The Power of Diversity: "Well-rounded" training on a mix of task types was highly effective. Interestingly, training on completely different task types (e.g., non-classification tasks) often helped as much as training on similar ones.
- MTL vs. Meta-Learning: Despite the theoretical advantages of meta-learning, simple MTL proved to be a remarkably strong baseline, often outperforming complex optimization loops. This suggests that for large transformers, exposure to task diversity is the primary driver of performance.
- Widespread Gains: Upstream learning improved few-shot performance on over 51% of test tasks by more than 5% relative gain. On the Random partition, models saw a 35% average improvement over direct fine-tuning.
- Catastrophic Forgetting: In some cases (about 12% of tasks), upstream learning actually hurt performance. Tasks like CoLA (linguistic acceptability) suffered, likely because the model "forgot" fundamental language modeling knowledge during the upstream stage.
Conclusion and Future Directions
CrossFit demonstrates that cross-task generalization is a viable path toward more robust few-shot learners. Future work will need to address "negative transfer" and explore automated ways to measure task similarity for better task selection. Additionally, integrating these generalization benefits with prompting techniques remains a promising frontier for NLP research.
Summary updated