Scaling Training Datasets Using Large Language Models

2,077 concepts1,470 papers47 proposals

Deep learning models are notoriously data-hungry. Historically, expanding a dataset meant hiring armies of human annotators or relying on rigid, deterministic heuristics like synonym substitution or random cropping. Large Language Models (LLMs) fundamentally change this paradigm. Instead of acting merely as end-user assistants, LLMs function as highly flexible, context-aware data factories. By leveraging their vast internal world knowledge and linguistic mastery, researchers can synthesize high-quality training data, balance skewed distributions, and construct complex evaluation benchmarks at a fraction of the traditional cost.

Bridging Structure and Natural Language

One of the primary reasons LLM-based data augmentation works so well is the model's ability to map structured data schemas to fluent natural language, and vice versa. Traditional template-based generators produce repetitive, easily overfitted text. LLMs, by contrast, inject stylistic variety and contextual realism.

For instance, in chart understanding, StructChart by Renqiu Xia et al. reformulates chart data into structured triplets and uses an LLM to generate diverse statistical datasets alongside code to render them into synthetic images. Similarly, Wei Ji et al. in Towards Complex-query Referring Image Segmentation: A Novel Benchmark utilize LLMs to synthesize complex, linguistically diverse natural language descriptions from structured object-relationship triplets for image segmentation tasks.

This bidirectional capability also works in reverse through a process known as "back-instruction." As demonstrated in Can Large Language Models Analyze Graphs like Professionals? A Benchmark, Datasets and Models, Weize Chen et al. use an API-driven back-instruction technique. This strategy starts with specific code API functions and prompts an LLM to generate corresponding natural language questions, ensuring comprehensive functional coverage for downstream training.

Controlled Transformations and Dataset Balancing

Beyond generating data from scratch, LLMs excel at transforming existing datasets to introduce targeted variations or correct systemic biases. Because an LLM understands semantic context, it can alter specific attributes of a sentence while preserving its core structural integrity.

A clear example of this is found in the work of Olha Nahurna in Gender Swapping as a Data Augmentation Technique: Developing Gender-Balanced Datasets for Ukrainian Language Processing. Here, an LLM automatically swaps gendered entities within sentences to create parallel, gender-balanced datasets for Ukrainian NLP. This semantic precision avoids the grammatical failures common to rule-based substitution engines.

Furthermore, LLMs can perform label-reversal transformations to balance datasets. In CoCa-CXR: Contrastive Captioners Learn Strong Temporal Structures for Chest X-Ray Vision-Language Understanding, Yixiong Chen et al. use an LLM to extract comparative statements from medical text and generate synthetic training pairs by swapping image order and flipping associated temporal labels (e.g., changing 'improved' to 'worsened'). In specialized domains like scientific literature extraction, Pranshu Rastogi et al. in Extracting Software Mentions and Relations using Transformers and LLM-Generated Synthetic Data at SOMD 2025 demonstrate that LLMs can merge and rephrase existing sentences while strictly preserving key tokens, significantly boosting Named Entity Recognition (NER) performance.

Elevating Complexity via Advanced Prompting and Filtering

A common skepticism regarding synthetic data is that it might inherit the hallucinations or superficial patterns of the generator. To combat this, modern frameworks do not simply ask an LLM for "more data"; they embed advanced reasoning strategies and execution-based verification loops into the synthesis pipeline.

In Zero-Shot and Efficient Clarification Need Prediction in Conversational Search, Lili Lu et al. employ chain-of-thought prompting and counterfactual generation strategies. By forcing the LLM to reason through the generation process and explicitly construct contrasting negative examples, the resulting synthetic data gains significantly higher discriminative power.

When dealing with long-form context, Chau Minh Pham et al. introduce CLIPPER: Compression enables long-context synthetic data generation, a framework that compresses lengthy documents into outlines and summaries before prompting the LLM. This compression allows the model to generate highly complex and valid narrative claims that span across long contexts without getting lost in the noise.

To guarantee absolute validity in rigid domains like mathematics, Sirui Li et al. in Towards Foundation Models for Mixed Integer Linear Programming combine evolutionary strategies with rigorous execution-based filtering. The LLM synthesizes mathematical optimization problems, which are then passed to an external solver; only structurally valid, executable problems are retained for the training set.

The Intuition of Knowledge Distillation and Privacy

Why does training a model on synthetic data work so well? At its core, LLM-based data augmentation acts as a form of generalized knowledge distillation. A massive, computationally expensive model (or its underlying pre-training corpus) compresses its broad understanding into a targeted, high-density dataset designed for a specific downstream task.

This allows smaller, specialized models to achieve remarkable performance gains. For example, Aritra Mitra in Assessing the Utility of C Comments with SVM and Naïve Bayes Classifier demonstrates that augmenting training sets with LLM-generated code-comment pairs provides a measurable boost to simpler baseline classifiers like SVMs.

Moreover, this paradigm solves critical deployment and privacy constraints. In Roll Up Your Sleeves: Working with a Collaborative and Engaging Task-Oriented Dialogue System, Lingbo Mo et al. use LLMs to generate synthetic training data for smaller, locally hosted models using intent descriptions and few-shot examples. This strategy circumvents data scarcity while maintaining strict user privacy and eliminating real-time LLM hallucinations during inference.

Finally, this scalability enables automated evaluation. José Pombal et al. introduce Zero-shot Benchmarking in Zero-shot Benchmarking: A Framework for Flexible and Scalable Automatic Evaluation of Language Models, leveraging a "meta-prompt" to dynamically generate diverse test instances and reference answers, creating highly adaptable benchmarks across languages and difficulties without manual intervention.

History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_llm_based_data_augmentation_168a232e,
  title = {LLM-based data augmentation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/llm-based-data-augmentation/rdr_168a232eb42a402c84b7a8ed9f9d3eba},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_llm_based_data_augmentation_168a232e_v1,
  title = {LLM-based data augmentation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/llm-based-data-augmentation/rdr_168a232eb42a402c84b7a8ed9f9d3eba/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake