Is GPT-3 a Good Data Annotator?

Bosheng Ding, Chengwei Qin, Linlin Liu, Yew Ken Chia, Boyang Li, Shafiq Joty, Lidong Bing

Introduction

Supervised machine learning is fundamentally "data-hungry," requiring thousands of accurately labeled examples to train specialized models like BERT. Traditionally, this process relies on human labor, which is expensive, slow, and difficult to scale, creating a significant bottleneck for smaller organizations and academic labs. This paper explores whether Large Language Models (LLMs) like GPT-3 can serve as effective data annotators, potentially democratizing AI development by providing high-quality labels at a fraction of the traditional cost.

Methodology

The researchers evaluated three distinct strategies for leveraging GPT-3's knowledge to create training data:

  1. Prompt-Guided Unlabeled Data Annotation (PGDA): GPT-3 acts as a labeler for existing unlabeled text.
  2. Prompt-Guided Training Data Generation (PGDG): GPT-3 generates both the text and the corresponding labels from scratch based on a prompt.
  3. Dictionary-Assisted Training Data Generation (DADG): GPT-3 uses external knowledge bases (e.g., Wikidata) to guide the generation of natural language sentences around specific entities.
Overview of the three data annotation and generation strategies using GPT-3: PGDA, PGDG, and DADG.

The resulting synthetic datasets were used to fine-tune a standard BERT-base model, which was then compared against models trained on human-annotated data across four tasks: sentiment analysis (SST2), relation extraction (FewRel), named entity recognition (CrossNER), and aspect sentiment triplet extraction (ASTE).

Key Insights

The study revealed that the optimal strategy depends on the task's "label space." For simple binary classification like sentiment analysis, PGDA is highly effective. However, for complex tasks with many categories (e.g., the 64 relations in FewRel), GPT-3 performs better when asked to generate examples for a specific label (PGDG) rather than choosing from a long list. This process effectively "distills" GPT-3's general knowledge into a smaller, production-ready model.

Experimental Results

The quantitative findings highlight a dramatic shift in the cost-benefit analysis of data annotation. On the SST2 sentiment task, a model trained on GPT-3-labeled data achieved 89.29% accuracy, coming remarkably close to the 93.52% achieved by human-labeled data.

Comparison of cost, time, and performance between GPT-3 strategies and human labeling on the SST2 dataset.

As shown in the results table, the efficiency gains are staggering. While human annotation for the full SST2 dataset cost between $4,800 and $6,700 and took weeks to manage, GPT-3 completed a comparable task for approximately $22 in just 27 minutes. In more complex tasks like FewRel, GPT-3-based models even outperformed those trained on smaller sets of human-labeled data.

Limitations and Future Work

Despite these promising results, several challenges remain. GPT-3 is a "black box," making it difficult to diagnose specific labeling errors or biases inherited from its training data. Furthermore, while significantly cheaper than humans, the costs of using proprietary LLMs can still be a factor for large-scale projects. Future research may focus on using open-source models like LLaMA or even more advanced models like GPT-4 to further refine these synthetic annotation pipelines.

Summary updated

BibTeX
@inproceedings{ding2023is,
  author = {Bosheng Ding and Chengwei Qin and Linlin Liu and Yew Ken Chia and Boyang Li and Shafiq Joty and Lidong Bing},
  title = {{Is GPT-3 a Good Data Annotator?}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
  doi = {10.18653/v1/2023.acl-long.626},
}
Figure