Learning Instructions with Unlabeled Data for Zero-Shot Cross-Task Generalization

Yuxian Gu, Pei Ke, Xiaoyan Zhu, Minlie Huang

Abstract

Instruction tuning (IT) has revolutionized the ability of large language models to generalize to unseen tasks. However, its reliance on massive, human-annotated datasets presents a significant bottleneck for low-resource domains. This paper introduces Unlabeled Data Augmented Instruction Tuning (UDIT), a framework that leverages heuristic rules to generate pseudo-labeled training data from abundant plain text. By pairing these pseudo-labels with human instructions, UDIT enables smaller models to achieve performance comparable to or exceeding much larger models trained on standard supervised data. Specifically, a 700M parameter model using UDIT outperforms a 3B parameter model trained with vanilla instruction tuning.

Introduction

The "instruction learning" paradigm involves fine-tuning models on a diverse collection of tasks described via human-written instructions. While effective for zero-shot generalization, this approach is data-hungry. Many niche tasks lack the multi-domain annotated examples required to prevent models from overfitting. While unlabeled plain text is cheap and plentiful, standard self-supervised objectives (like masked word prediction) fail to teach models the complex mapping between explicit human instructions (e.g., "Summarize this text") and task semantics. UDIT addresses this by using programmatic heuristics to bridge the gap between unlabeled text and instruction-following capabilities.

The UDIT Framework

Overview of the UDIT framework showing how plain text is converted into pseudo-labeled tasks like summarization and QA for instruction tuning.

The core of UDIT is the construction of pseudo-labeled data from plain text using simple programmatic rules. These rules format text into task-like structures:

  • Summarization: Uses an article's title as the target summary and the body as input.
  • Multiple-Choice QA: Randomly replaces a noun with a blank to create a question, using surrounding text as context and other sampled nouns as distractors.
  • Topic Classification: Extracts keywords from source URLs (e.g., "sports" from a news link) to serve as labels for the article text.

Once generated, this pseudo-labeled data is mixed with human instructions and used for multi-task text-to-text training. This allows the model to practice the mapping between instruction formats and task semantics without requiring additional human annotation.

Experimental Setup

The authors utilized a 37.2GB multi-domain corpus (including Wikipedia and BookCorpus) for unlabeled text. Training involved 8 task clusters from the T0 benchmark, comprising 36 datasets and 304 instructions. Evaluation focused on zero-shot cross-task generalization across unseen clusters such as Coreference Resolution, Natural Language Inference (NLI), and Sentence Completion.

Key Results and Insights

Performance comparison showing UDIT 700M surpassing the T0 3B baseline as labeled data increases.

UDIT demonstrates exceptional data efficiency and performance gains:

  • Scaling Efficiency: A 700M parameter T5 model trained with UDIT achieves an average accuracy of 57.8%, surpassing a 3B parameter Vanilla-IT model (57.2%).
  • Zero-Label Performance: In a scenario with zero human-labeled training data (using only instructions and plain text), UDIT reaches 49.6% accuracy, significantly higher than the direct zero-shot baseline of 43.0%.
  • Domain Diversity: The success of UDIT is tied to the diversity of the unlabeled text. Restricting data to a single domain (like Wikipedia) reduces gains, suggesting that pseudo-labeled data acts as a vital regularizer against domain overfitting.
  • Robustness: Despite the inherent noise in programmatically generated labels, the model remains robust, as the volume and diversity of tasks compensate for individual errors.

Conclusion and Future Work

UDIT provides a scalable path for instruction tuning by tapping into the vast reserves of unlabeled text. While current heuristics are manually designed, they prove that practicing the "instruction-to-task" mapping is more critical than pure language modeling for generalization. Future research will focus on automating heuristic discovery, mining instructions directly from corpora, and developing advanced denoising techniques for pseudo-labels to further push the performance ceiling.

Summary updated

Abstract

Training language models to learn from human instructions for zero-shot cross-task generalization has attracted much attention in NLP communities. Recently, instruction tuning (IT), which fine-tunes a pre-trained language model on a massive collection of tasks described via human-craft instructions, has been shown effective in instruction learning for unseen tasks. However, IT relies on a large amount of human-annotated samples, which restricts its generalization. Unlike labeled data, unlabeled data are often massive and cheap to obtain. In this work, we study how IT can be improved with unlabeled data. We first empirically explore the IT performance trends versus the number of labeled data, instructions, and training tasks. We find it critical to enlarge the number of training instructions, and the instructions can be underutilized due to the scarcity of labeled data. Then, we propose Unlabeled Data Augmented Instruction Tuning (UDIT) to take better advantage of the instructions during IT by constructing pseudo-labeled data from unlabeled plain texts. We conduct extensive experiments to show UDIT's effectiveness in various scenarios of tasks and datasets. We also comprehensively analyze the key factors of UDIT to investigate how to better improve IT with unlabeled data. The code is publicly available at https://github.com/thu-coai/UDIT.
BibTeX
@inproceedings{gu2022learning,
  author = {Yuxian Gu and Pei Ke and Xiaoyan Zhu and Minlie Huang},
  title = {{Learning Instructions with Unlabeled Data for Zero-Shot Cross-Task Generalization}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2022},
  doi = {10.18653/v1/2022.emnlp-main.105},
}
Figure