CrossFit: A Few-shot Learning Challenge for Cross-task Generalization in NLP

Qinyuan Ye, Bill Yuchen Lin, Xiang Ren

Abstract

Humans can learn a new language task efficiently with only few examples, by leveraging their knowledge obtained when learning prior tasks. In this paper, we explore whether and how such cross-task generalization ability can be acquired, and further applied to build better few-shot learners across diverse NLP tasks. We introduce CrossFit, a problem setup for studying cross-task generalization ability, which standardizes seen/unseen task partitions, data access during different learning stages, and the evaluation protocols. To instantiate different seen/unseen task partitions in CrossFit and facilitate in-depth analysis, we present the NLP Few-shot Gym, a repository of 160 diverse few-shot NLP tasks created from open-access NLP datasets and converted to a unified text-to-text format. Our analysis reveals that the few-shot learning ability on unseen tasks can be improved via an upstream learning stage using a set of seen tasks. We also observe that the selection of upstream learning tasks can significantly influence few-shot performance on unseen tasks, asking further analysis on task similarity and transferability.

Introduction

Standard few-shot learning in NLP usually focuses on instance-level generalization: if a model is given 16 examples of sentiment analysis, can it predict the 17th? While effective, this ignores how humans actually learn. We don't learn every task in a vacuum; we use our experience with reading comprehension to help us with summarization, or our logic from natural language inference to help with fact-checking.

This is cross-task generalization. The challenge is that NLP tasks are incredibly fragmented—different formats, different labels, and different evaluation metrics. Most existing research stays within narrow silos (e.g., only looking at classification). CrossFit aims to break these silos by creating a standardized "gym" where models must learn to adapt to entirely unseen types of tasks by practicing on a wide variety of "seen" ones.

Methods

The authors propose a two-stage learning framework to acquire and evaluate cross-task skills:

  1. Upstream Learning: The model is trained on a large set of "seen" tasks ($T_{train}$). The goal is to acquire a general ability to learn new tasks quickly.
  2. Downstream Fine-tuning: The model is evaluated on "unseen" tasks ($T_{test}$) using only a few examples (e.g., 16 or 32).
The CrossFit challenge framework: Upstream learning on seen tasks followed by downstream fine-tuning on unseen tasks.

Key Innovations

  • Unified Format: Following the T5 paradigm, every task in the gym is converted into a text-to-text format. Whether it’s a multiple-choice question or a regression task, the model always sees a string and produces a string.
  • Average Relative Gain (ARG): Since tasks use different metrics (F1, Accuracy, ROUGE), the authors use ARG to measure the percentage improvement a model gains from its upstream experience compared to a baseline that starts from scratch.
  • Algorithms: They compare Multi-task Learning (MTL)—simply training on all seen tasks simultaneously—against popular Meta-learning algorithms like MAML, First-order MAML, and Reptile, which are explicitly designed for "learning to learn."

Datasets: The NLP Few-shot Gym

The core contribution is the NLP Few-shot Gym, which includes 160 diverse tasks curated from HuggingFace Datasets.

  • Task Variety: Includes sentiment analysis, question answering (reading comprehension, multiple-choice, closed-book), summarization, dialogue, and even regression.
  • Partitions: The authors designed eight different splits to test specific scenarios, such as "Random Split" (general diversity) and "Held-out-NLI" (can a model learn NLI if it has never seen an NLI task before?).
Standardized task partitions in CrossFit, including random splits and held-out task categories.

Key Results and Insights

  • MTL is a Strong Baseline: Surprisingly, simple multi-task learning frequently outperformed sophisticated meta-learning algorithms like MAML. On the Random Partition, the MTL approach achieved an ARG of 35.06%, significantly outperforming the best meta-learning approach (MAML at 28.50%).
  • The "Well-Rounded" Advantage: Training on non-classification tasks was found to be just as helpful for classification performance as training on other classification tasks. This suggests that the model is learning general linguistic "skills" rather than just task-specific patterns.
  • Catastrophic Forgetting of Pre-training: For some tasks (like CoLA), upstream learning actually hurt performance. The authors hypothesize that the upstream tasks might cause the model to "forget" the specific denoising/masked language modeling knowledge it gained during its initial pre-training (like BART or T5).
  • Data Scaling Limits: The diversity of tasks seems more important than the depth of data per task. Giving the model more data per upstream task didn't consistently lead to better generalization on unseen tasks.
  • Overall Boost: Upstream learning improved performance on 51.47% of test tasks by more than 5% relative to direct fine-tuning.

Conclusion

The CrossFit challenge demonstrates that leveraging a diverse set of prior tasks can significantly enhance a model's ability to learn new tasks from few examples. While simple multi-task learning proves to be a formidable baseline, the issues of negative transfer and catastrophic forgetting of pre-trained knowledge remain open research questions. Future work will focus on automated task selection and the development of meta-learning algorithms better suited for large-scale transformer architectures.

Summary updated

Abstract

Humans can learn a new language task efficiently with only few examples, by leveraging their knowledge obtained when learning prior tasks. In this paper, we explore whether and how such cross-task generalization ability can be acquired, and further applied to build better few-shot learners across diverse NLP tasks. We introduce CrossFit, a problem setup for studying cross-task generalization ability, which standardizes seen/unseen task partitions, data access during different learning stages, and the evaluation protocols. To instantiate different seen/unseen task partitions in CrossFit and facilitate in-depth analysis, we present the NLP Few-shot Gym, a repository of 160 diverse few-shot NLP tasks created from open-access NLP datasets and converted to a unified text-to-text format. Our analysis reveals that the few-shot learning ability on unseen tasks can be improved via an upstream learning stage using a set of seen tasks. We also observe that the selection of upstream learning tasks can significantly influence few-shot performance on unseen tasks, asking further analysis on task similarity and transferability.
BibTeX
@inproceedings{ye2021crossfit,
  author = {Qinyuan Ye and Bill Yuchen Lin and Xiang Ren},
  title = {{CrossFit: A Few-shot Learning Challenge for Cross-task Generalization in NLP}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2021},
  doi = {10.18653/v1/2021.emnlp-main.572},
}
Figure