Knowledge Distillation in Automated Annotation: Supervised Text Classification with LLM-Generated Training Labels

Nicholas Pangakis, Sam WolkenNAACL 2024

TL;DR

This paper explores using Large Language Models (LLMs) like GPT-4 to generate training labels for supervised text classifiers, finding that models trained on LLM-generated labels perform comparably to those trained on human labels. This offers a faster, cheaper, and more efficient way to build text classification systems, especially for computational social science research.

What Problem Does This Paper Address?

Computational social scientists often rely on supervised text classifiers to analyze large volumes of text data. However, creating the human-labeled datasets needed to train these classifiers is a time-consuming, expensive, and error-prone process. While LLMs show promise for automated annotation, their few-shot capabilities can still be costly for large datasets. This paper investigates whether LLM-generated labels can be effectively used to fine-tune supervised models, potentially offering a more cost-effective and efficient alternative to full human annotation or expensive few-shot LLM inference.

Methods

The authors propose a four-step workflow:

  1. Validate LLM Few-Shot Performance: They first assess how well an LLM (GPT-4) performs few-shot annotation on a small subset of human-labeled data. This is crucial because LLM performance can vary.
  2. LLM Generates Training Labels: Using a validated prompt, the LLM then labels a larger subset of text samples (1,000 per task) to serve as training data.
  3. Train Supervised Models: Standard supervised text classifiers (BERT, RoBERTa, DistilBERT) are then fine-tuned using these LLM-generated labels.
  4. Test Model Performance: The performance of these fine-tuned models is evaluated on a held-out set of human-labeled data.

They also explore knowledge distillation as a potential avenue for approximating teacher model performance with smaller student models, though the primary focus is on using LLM-generated labels for fine-tuning.

Datasets

The study utilizes a novel corpus of 14 classification tasks derived from recent computational social science articles published in high-impact journals. These datasets are stored in password-protected archives, which the authors highlight as a key feature to mitigate contamination risks (i.e., preventing the LLM from having seen the exact training data during its pre-training). The human-labeled ground truth data from these original studies is used for validation and testing.

Key Insights and Ideas

  • LLM Labels as a Viable Alternative: The core finding is that supervised models fine-tuned on LLM-generated labels perform nearly as well as models trained on human-annotated data. This suggests LLMs can significantly streamline the annotation process without a drastic drop in performance.
  • Cost and Efficiency Gains: Using LLM-generated labels for fine-tuning is presented as a fast, efficient, and cost-effective method, potentially reducing annotation costs by over 90% compared to traditional human annotation for large datasets.
  • Recall vs. Precision Trade-off: Models trained on LLM-generated labels tend to excel in recall but may be slightly weaker in precision compared to human-labeled models. This implies LLM-generated labels might be particularly suitable for tasks where identifying all relevant instances (high recall) is more critical than ensuring every identified instance is correct (high precision).
  • Human Validation Remains Crucial: Despite the effectiveness of LLM-generated labels, the authors strongly emphasize the necessity of human validation. LLMs can still produce inaccurate labels, and downstream models will inherit any biases or poor performance from these initial labels.

Key Results

  • Comparable Performance: The median F1 performance gap between models fine-tuned on LLM-generated labels and those fine-tuned on human labels was only 0.039.
  • Close to Few-Shot LLMs: Supervised models fine-tuned on GPT-4 generated labels performed remarkably close to GPT-4 few-shot models, with a median F1 difference of just 0.006.
  • Recall Advantage: GPT-4 few-shot models and supervised models fine-tuned on GPT-4 labels showed significantly better median recall than models trained on human labels.
  • Precision Disadvantage: Conversely, human-labeled models achieved higher precision. For example, BERT fine-tuned on human labels had 0.214 higher median precision than BERT fine-tuned on GPT-4 labels.

Limitations and Future Work

  • Data Contamination: While efforts were made to mitigate contamination by using private archives, the authors acknowledge that LLMs might have encountered similar data during pre-training, potentially inflating performance metrics.
  • LLM Inaccuracies and Biases: The paper notes that LLM-generated labels can still be inaccurate or biased, and downstream models will inherit these issues. Human validation is essential to catch these errors.
  • Model Drift: The authors conducted an ablation study to check for performance changes over time, finding minimal drift in GPT-4's performance over a six-month period.
  • Noise Robustness: An ablation study indicated that supervised models are fairly robust to noisy LLM-generated labels, with minimal performance differences observed between models trained on clean vs. noisy labels.
  • Future Work: The authors suggest that future research could explore more advanced prompt engineering techniques, investigate the impact of different LLM architectures, and develop more automated methods for prompt tuning and validation. They also highlight the importance of human-centered evaluation standards for LLM applications in annotation.

Summary updated

BibTeX
@inproceedings{pangakis2024knowledge,
  author = {Nicholas Pangakis and Sam Wolken},
  title = {{Knowledge Distillation in Automated Annotation: Supervised Text Classification with LLM-Generated Training Labels}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2024},
}
Figure