Fabricator: An Open Source Toolkit for Generating Labeled Training Data with Teacher LLMs

Jonas Golde, Patrick Haller, Felix Hamborg, Julian Risch, Alan AkbikEMNLP 2023

TL;DR

This paper introduces FABRICATOR, an open-source Python toolkit designed to simplify and standardize the process of generating labeled training data using large language models (LLMs). It addresses the bottleneck of manual data annotation by enabling researchers to leverage LLMs as "teachers" to create datasets for various NLP tasks, making dataset generation more accessible and reproducible.

What Problem Does This Paper Address?

Training effective Natural Language Processing (NLP) models, especially state-of-the-art ones, typically requires large amounts of high-quality labeled data. However, manually creating this data is a time-consuming, expensive, and often domain-expertise-heavy process. This bottleneck limits the development and application of NLP models, particularly in low-resource scenarios or for specialized tasks. While the concept of using LLMs to generate synthetic data (zero-shot learning via dataset generation) shows promise, there's a lack of accessible, standardized tools to facilitate this research and its practical application.

Methods

FABRICATOR provides a modular framework for dataset generation, orchestrating the interaction between LLMs, user-defined prompts, and existing datasets. Its core components include:

  • DatasetGenerator: The central orchestrator that manages the generation process, handling prompt composition, LLM calls, and output formatting.
  • BasePrompt: A flexible class for defining prompts, allowing users to specify task descriptions, label options, and integrate few-shot examples. It supports complex prompting scenarios, including those for tasks like textual similarity and entity recognition.
  • LLM Integration: FABRICATOR leverages the HAYSTACK framework's PromptNode for seamless integration with various LLMs, whether API-based (OpenAI, Anthropic, etc.) or self-hosted.
  • HuggingFace Interoperability: The generated datasets are outputted as HuggingFace Dataset objects, allowing for easy integration with existing NLP training pipelines and sharing on the HuggingFace Hub.

FABRICATOR supports three main generation workflows:

  1. Generate unlabeled data: Creating raw text data for tasks like question answering or topic modeling.
  2. Generate label-conditioned data: Producing data for classification tasks by prompting the LLM to generate text for specific labels.
  3. Annotate unlabeled data: Adding labels to existing unlabeled text corpora.

Datasets

The paper uses several standard NLP datasets for evaluation and demonstration:

  • IMDB: A binary sentiment classification benchmark.
  • MRPC: A paraphrase detection dataset.
  • SNLI: A dataset for natural language inference.
  • TREC-6: A question type categorization dataset.
  • SQuAD-v2: A dataset for extractive question answering.
  • CoNLL-03: A dataset for named entity recognition (token classification).

The paper does not introduce new benchmark datasets but rather demonstrates how FABRICATOR can be used to generate synthetic data for these existing benchmarks.

Key Insights and Ideas

  • LLMs as data generators are powerful but task-dependent: While LLMs can generate data that leads to comparable performance to human-annotated data for simpler tasks like binary sentiment classification (IMDB), their effectiveness diminishes for more complex tasks like multi-class classification (SNLI) or extractive question answering (SQuAD).
  • Few-shot examples significantly improve generation quality: Including even a small number of few-shot examples in prompts can lead to substantial improvements in the performance of models trained on the generated data, especially for tasks like TREC-6.
  • Prompt engineering is crucial, but overloading is detrimental: There's a sweet spot for the number of few-shot examples to include in a prompt. Including too many examples can sometimes lead to performance degradation.
  • Instruction-tuned LLMs show promise for complex tasks: Early results suggest that instruction-tuned open-source LLMs (like LLaMA) might offer better annotation quality for complex tasks compared to general-purpose LLMs like GPT-3.5, potentially closing the gap with human-annotated data.

Key Results

  • For IMDB (binary sentiment classification), models trained on FABRICATOR-generated data achieved accuracy within 1.0 percentage point of models trained on gold-annotated data, using 10k generated examples.
  • For more complex tasks like SNLI and SQuAD, performance gaps of 19.0 pp and 16.3 pp respectively were observed when comparing models trained on generated vs. human-annotated data (with 10k examples).
  • In TREC-6, increasing the number of few-shot examples used in prompts led to significant improvements, with some settings outperforming the baseline by over 24 percentage points in accuracy.
  • For CoNLL-03, instruction-tuned LLaMA models showed higher F1 scores than GPT-3.5 for annotation, suggesting potential for better performance on complex tasks.

Limitations and Future Work

  • Limited task coverage in evaluation: The authors acknowledge that their evaluation covered only a subset of NLP tasks due to resource constraints. Future work should expand this to a broader range of tasks.
  • Repository applicability: While designed to be versatile, there might be specific tasks or domains where the current repository structure or features may not be directly applicable.
  • Quality and diversity metrics: The authors aim to include research on quality and diversity metrics for generated data and encourage community contributions to address these.
  • Prompt optimization: The experiments, particularly in the appendix, highlight that prompt optimization and hyperparameter tuning were not extensively performed, suggesting room for improvement.
  • Ethical considerations: The paper raises concerns about potential hallucination and biases in LLM-generated data, emphasizing the need for careful validation and bias detection. Future work should focus on developing methods to mitigate these issues.

Summary updated

BibTeX
@inproceedings{golde2023fabricator,
  author = {Jonas Golde and Patrick Haller and Felix Hamborg and Julian Risch and Alan Akbik},
  title = {{Fabricator: An Open Source Toolkit for Generating Labeled Training Data with Teacher LLMs}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2023},
}
Figure