Abstract
Instruction tuning (IT) has revolutionized the ability of large language models to generalize to unseen tasks. However, its reliance on massive, human-annotated datasets presents a significant bottleneck for low-resource domains. This paper introduces Unlabeled Data Augmented Instruction Tuning (UDIT), a framework that leverages heuristic rules to generate pseudo-labeled training data from abundant plain text. By pairing these pseudo-labels with human instructions, UDIT enables smaller models to achieve performance comparable to or exceeding much larger models trained on standard supervised data. Specifically, a 700M parameter model using UDIT outperforms a 3B parameter model trained with vanilla instruction tuning.
Introduction
The "instruction learning" paradigm involves fine-tuning models on a diverse collection of tasks described via human-written instructions. While effective for zero-shot generalization, this approach is data-hungry. Many niche tasks lack the multi-domain annotated examples required to prevent models from overfitting. While unlabeled plain text is cheap and plentiful, standard self-supervised objectives (like masked word prediction) fail to teach models the complex mapping between explicit human instructions (e.g., "Summarize this text") and task semantics. UDIT addresses this by using programmatic heuristics to bridge the gap between unlabeled text and instruction-following capabilities.
The UDIT Framework
The core of UDIT is the construction of pseudo-labeled data from plain text using simple programmatic rules. These rules format text into task-like structures:
- Summarization: Uses an article's title as the target summary and the body as input.
- Multiple-Choice QA: Randomly replaces a noun with a blank to create a question, using surrounding text as context and other sampled nouns as distractors.
- Topic Classification: Extracts keywords from source URLs (e.g., "sports" from a news link) to serve as labels for the article text.
Once generated, this pseudo-labeled data is mixed with human instructions and used for multi-task text-to-text training. This allows the model to practice the mapping between instruction formats and task semantics without requiring additional human annotation.
Experimental Setup
The authors utilized a 37.2GB multi-domain corpus (including Wikipedia and BookCorpus) for unlabeled text. Training involved 8 task clusters from the T0 benchmark, comprising 36 datasets and 304 instructions. Evaluation focused on zero-shot cross-task generalization across unseen clusters such as Coreference Resolution, Natural Language Inference (NLI), and Sentence Completion.
Key Results and Insights
UDIT demonstrates exceptional data efficiency and performance gains:
- Scaling Efficiency: A 700M parameter T5 model trained with UDIT achieves an average accuracy of 57.8%, surpassing a 3B parameter Vanilla-IT model (57.2%).
- Zero-Label Performance: In a scenario with zero human-labeled training data (using only instructions and plain text), UDIT reaches 49.6% accuracy, significantly higher than the direct zero-shot baseline of 43.0%.
- Domain Diversity: The success of UDIT is tied to the diversity of the unlabeled text. Restricting data to a single domain (like Wikipedia) reduces gains, suggesting that pseudo-labeled data acts as a vital regularizer against domain overfitting.
- Robustness: Despite the inherent noise in programmatically generated labels, the model remains robust, as the volume and diversity of tasks compensate for individual errors.
Conclusion and Future Work
UDIT provides a scalable path for instruction tuning by tapping into the vast reserves of unlabeled text. While current heuristics are manually designed, they prove that practicing the "instruction-to-task" mapping is more critical than pure language modeling for generalization. Future research will focus on automating heuristic discovery, mining instructions directly from corpora, and developing advanced denoising techniques for pseudo-labels to further push the performance ceiling.
Summary updated