Data-Efficient Finetuning Using Cross-Task Nearest Neighbors

Hamish Ivison, Noah A. Smith, Hannaneh Hajishirzi, Pradeep Dasigi

TL;DR

DEFT (Data-Efficient FineTuning) is a novel method that uses a handful of unlabeled examples from a target task to "cherry-pick" the most relevant training data from a massive multitask pool. By training only on these semantic neighbors, DEFT outperforms models trained on the entire pool while using up to 50x less data, effectively mitigating the performance-degrading effects of negative task interference.

What Problem Does This Paper Address?

The current paradigm for building general-purpose language models involves training on massive, diverse multitask mixtures (like T0 or FLAN). While this promotes zero-shot generalization, it faces two major hurdles:

  1. Negative Interference: Training on a huge variety of tasks can be counterproductive; unrelated tasks can "interfere" with each other, leading to worse performance on specific target domains compared to a more focused model.
  2. Data Inefficiency: Training on millions of examples is computationally expensive. Furthermore, obtaining high-quality labeled data for a new, niche task (like legal or scientific QA) is often prohibitively costly.

The authors ask: Can we build a better task-specific model by using only the most relevant parts of a large multitask pool?

Methods

The authors propose DEFT, a three-step pipeline illustrated in Figure 1:

Figure 1: Overview of the DEFT method. Given some unlabeled target-task instances, we find the most similar instances in a large pool of multitask data. We train a model on these instances. If we have access to labeled data, we optionally few-shot finetune the DEFT model.
  1. Indexing: They take a massive pool of prompted multitask data (like P3, which contains ~100 million instances) and encode every instance into a dense vector using a pretrained T5 encoder. These vectors are stored in a searchable index (FAISS).
  2. Retrieval: Given a small set (32–1000) of unlabeled target-task examples, they retrieve the $k$-nearest neighbors from the index. This creates a custom training subset of "cross-task neighbors" that are semantically similar to the target task.
  3. Finetuning: A pretrained language model is then finetuned only on this retrieved subset.

Crucially, DEFT selects data at the instance level rather than the dataset level. This means it can pull relevant examples from many different source tasks, even if the source tasks as a whole seem unrelated to the target.

Datasets

  • Multitask Pools: The primary pool is P3 (Public Pool of Prompts), used to train the T0 model. They also experiment with SuperNaturalInstructions (SNI), a collection of 1,600+ tasks with natural language instructions.
  • Evaluation Tasks: They test on 11 standard NLU datasets (RTE, ANLI, HellaSwag, etc.) and 3 challenging "out-of-domain" datasets: CaseHold (legal QA), DROP (reading comprehension with discrete reasoning), and QasperEvidence (scientific paper QA).

Key Insights and Ideas

  • Less is More: Training on just 2% of the P3 pool (carefully selected via DEFT) consistently beat training on the full 100% mixture. This is a strong piece of evidence that massive multitask training suffers from significant negative interference.
  • Unlabeled Data as a Compass: You don't need labels to find good training data. The semantic similarity of the inputs alone is enough to identify which source examples will help the model learn the right skills for the target task.
  • Instance-Level Granularity is Key: The authors found that for any given target task, only a small fraction of instances from any single source dataset are actually retrieved. This suggests that traditional "dataset selection" methods are too coarse.
  • Better Starting Blocks: A model trained via DEFT provides a much better initialization for subsequent few-shot learning than a general-purpose multitask model like T0.

Key Results

  • Zero-Shot Superiority: DEFT-XL (3B parameters) achieved a 13% median relative improvement over T0-3B across 14 datasets, despite T0 being trained on significantly more data.
  • Domain Adaptation: On specialized tasks like QasperEvidence (scientific QA), where T0-3B performs near random chance, DEFT-XL showed dramatic gains (28.5 F1 vs 19.9 F1).
  • Few-Shot Gains: When combined with parameter-efficient finetuning (IA3), DEFT models outperformed few-shot finetuned T0-3B models by 2–23% on 8 out of 11 datasets.
  • Cost-Effectiveness: As shown in Figure 3, for low annotation budgets, it is much more effective to collect a few hundred unlabeled examples for DEFT retrieval than to spend that same budget on a handful of labeled examples for standard finetuning.
Figure 3: Performance of DEFT-XL and full-finetuning methods with the same annotation budget used for obtaining either labeled or unlabeled data for QasperEvidence. Unless one has a large annotation budget, collecting unlabeled examples is superior to collecting labelled ones. |R| and |T| refer to the size of the retrieval and train sets respectively. See Section 5.2 for details.

Limitations and Future Work

  • Not "Pure" Zero-Shot: DEFT requires access to at least some unlabeled data from the target task, which might not always be available in a streaming or truly open-ended setting.
  • Index Overhead: While the training phase is highly efficient, building the initial dense index for a 100-million-instance pool is a significant one-time computational cost.
  • Task Diversity: The authors noted that while similarity is a great signal, maintaining some level of task diversity in the retrieved set is important for stability, a factor DEFT does not yet explicitly optimize for. Future work could explore balancing similarity with diversity to further boost robustness.

Summary updated

Abstract

Obtaining labeled data to train a model for a task of interest is often expensive. Prior work shows training models on multitask data augmented with task descriptions (prompts) effectively transfers knowledge to new tasks. Towards efficiently building task-specific models, we assume access to a small number (32-1000) of unlabeled target-task examples and use those to retrieve the most similar labeled examples from a large pool of multitask data augmented with prompts. Compared to the current practice of finetuning models on uniformly sampled prompted multitask data (e.g.: FLAN, T0), our approach of finetuning on cross-task nearest neighbors is significantly more data-efficient. Using only 2% of the data from the P3 pool without any labeled target-task data, our models outperform strong baselines trained on all available data by 3-30% on 12 out of 14 datasets representing held-out tasks including legal and scientific document QA. Similarly, models trained on cross-task nearest neighbors from SuperNaturalInstructions, representing about 5% of the pool, obtain comparable performance to state-of-the-art models on 12 held-out tasks from that pool. Moreover, the models produced by our approach also provide a better initialization than single multitask finetuned models for few-shot finetuning on target-task data, as shown by a 2-23% relative improvement over few-shot finetuned T0-3B models on 8 datasets.
BibTeX
@inproceedings{ivison2023data,
  author = {Hamish Ivison and Noah A. Smith and Hannaneh Hajishirzi and Pradeep Dasigi},
  title = {{Data-Efficient Finetuning Using Cross-Task Nearest Neighbors}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
  doi = {10.18653/v1/2023.findings-acl.576},
}
Figure