Abstract
Training large language models on every multitask dataset available is computationally expensive and can lead to negative interference, where irrelevant tasks degrade performance on a specific target. This paper introduces DEFT (Data-Efficient FineTuning), a method that leverages a small set of unlabeled target-task examples to retrieve only the most relevant labeled instances from a massive multitask pool. By training on approximately 2% of the available data, DEFT consistently outperforms models trained on the entire pool, demonstrating that targeted data selection is superior to the "kitchen sink" approach of instruction tuning.
Introduction
The paradigm of instruction tuning—training on a diverse collection of prompted tasks—has significantly improved the zero-shot generalization of large language models (LLMs). However, this approach faces two primary challenges: the immense computational cost of training on millions of examples and the risk of negative interference, where dissimilar tasks conflict during the learning process.
The authors of this paper propose a more surgical approach. Instead of training on the entire multitask pool, they ask whether a model can achieve better performance by focusing solely on data relevant to a specific target task. Crucially, they aim to identify this relevant subset without requiring expensive human-labeled examples for the new task.
Methods: The DEFT Framework
DEFT (Data-Efficient FineTuning) introduces an elegant workflow that avoids the need for target-task labels during the data selection phase. The process consists of four main stages:
- Indexing: A massive pool of prompted multitask data (such as P3 or SuperNaturalInstructions) is encoded into a vector space using a pretrained encoder (e.g., T5).
- Retrieval: Given a small set (32 to 1,000) of unlabeled examples from a target task, the system performs a $k$-nearest neighbor search to identify the most semantically similar labeled examples in the multitask pool.
- Targeted Finetuning: A pretrained model is finetuned exclusively on this retrieved subset.
- Optional Few-Shot Tuning: If a small number of labeled examples are available for the target task, a final round of parameter-efficient finetuning (using the IA3 method) is performed to create "DEFT-Few."
A key innovation of DEFT is its instance-level selection. Unlike methods that select entire datasets, DEFT can pull specific, relevant examples from diverse sources, allowing it to "mine" a general pool for domain-specific knowledge even if the source datasets are not perfectly aligned with the target.
Experimental Setup
The researchers evaluated DEFT using two primary source pools:
- P3 (Public Pool of Prompts): ~100 million examples used to train the T0 model.
- SuperNaturalInstructions (SNI): 1,600+ tasks with expert-written instructions.
The target tasks included the 11 datasets used to evaluate T0 (e.g., RTE, ANLI, WinoGrande) and three specialized "diverse domain" datasets: CaseHold (legal), DROP (discrete reasoning), and Qasper (scientific NLP).
Results and Discussion
The experimental results demonstrate that "less is more" in the context of finetuning. DEFT-XL (3B parameters) outperformed the standard T0-3B model on 12 out of 14 datasets, achieving a median relative improvement of 13% while using only about 2% of the training data from the P3 pool.
Key Findings
- Mitigating Negative Interference: The superiority of DEFT over full-pool training confirms that massive multitask pools contain significant "noise" for any specific task. Filtering for similarity effectively removes interfering data.
- The Power of Unlabeled Data: Semantic similarity between inputs is a sufficient signal for data selection. This makes DEFT highly practical, as unlabeled data is often readily available.
- Superior Initializations: DEFT models provide a better starting point for few-shot learning than general multitask models. When 20-70 labels were provided, DEFT-Few performed 7% better on average than T0-3B.
- Domain Specialization: DEFT showed significant gains in specialized domains like legal (CaseHold) and scientific (Qasper) text, proving its ability to extract relevant knowledge from general-purpose pools.
Limitations and Future Work
While effective, DEFT has limitations. It requires at least a small set of unlabeled examples from the target task, which may not always be available. Furthermore, the current retrieval method focuses purely on similarity; incorporating diversity into the selection process could prevent the model from seeing redundant examples. Finally, the effectiveness of DEFT depends on the quality and coverage of the source pool; if the pool lacks relevant data, retrieval-based finetuning will be limited.
Future research could explore scaling DEFT to much larger models (70B+ parameters) and refining retrieval strategies to better balance similarity and diversity.
Summary updated