Does Training on Synthetic Data Make Models Less Robust?

Lingze Zhang, Ellie Pavlick

Abstract

As high-quality human-generated data becomes increasingly scarce, the AI industry is turning toward synthetic data generated by large language models (LLMs) for training. This study investigates whether this practice exacerbates model "blindspots"—the tendency to rely on superficial heuristics rather than genuine reasoning. Using Natural Language Inference (NLI) as a case study, researchers evaluated the robustness of models trained on synthetic versus original human data. The findings suggest that while synthetic data does not inherently resolve robustness issues, it also does not consistently degrade them more than human data, provided the synthetic data is not intentionally biased.

Introduction

LLMs are known to utilize ungeneralizable heuristics, or "shortcuts," to solve complex tasks. In Natural Language Inference (NLI), for instance, models often assume that high lexical overlap between a premise and a hypothesis implies entailment. This research addresses a critical concern: does training a model on its own or a peer's synthetic output create a feedback loop that reinforces these bad habits, ultimately making models less robust and more prone to heuristic-based errors?

Methodology

The researchers utilized the Llama-2-7B-hf model in two distinct roles: a Generator ($G$) to produce synthetic training sets and a Task Model ($T$) for training and evaluation. The experimental pipeline involved:

  1. Initial Training: Creating six "starting point" models by fine-tuning Llama-2 on varying amounts of the original MultiNLI dataset (0 to ~392k examples).
  2. Data Generation: Fine-tuning a separate Llama-2 model to generate approximately 1.8 million synthetic NLI examples.
  3. Augmentation: Further fine-tuning the starting point models on three dataset types: Original (human-generated), Synthetic (randomly sampled model-generated), and Biased Synthetic (filtered to reinforce lexical overlap heuristics).
  4. Evaluation: Testing models on the general MultiNLI dataset and the HANS (Heuristic Analysis for NLI Systems) dataset, which is specifically designed to break models relying on lexical shortcuts.

Results and Analysis

General Performance vs. Robustness

Surprisingly, for undertrained models, synthetic data improved general performance on MultiNLI almost as effectively as real human data. However, a "robustness gap" exists; while synthetic data didn't necessarily improve HANS performance, it didn't cause the catastrophic collapse initially hypothesized unless the data was intentionally biased.

Accuracy on MNLI and HANS across different starting points and augmentation sizes.

As shown in the figure above, the expected trend—where general accuracy increases while robustness decreases—only appeared clearly in specific scenarios, such as the 20k-example starting point. In many cases, the model's robustness remained relatively stable despite the shift to synthetic training inputs.

Lexical Overlap Statistics

Analysis of the datasets revealed that synthetic data does indeed contain more lexical overlap than human data (3.8% vs 1.6% for entailment labels). Despite this increase, the difference was not substantial enough to trigger a total loss of robustness in the fine-tuned models.

Comparison of lexical overlap (LO) statistics between synthetic and original datasets.

Conversely, the "Biased Synthetic" dataset, where lexical overlap was artificially forced to correlate perfectly with entailment, caused HANS performance to crash toward zero. This confirms that while unfiltered synthetic data is relatively safe, intentional or systemic bias in synthetic generation poses a significant risk.

Conclusion

This study demonstrates that synthetic data is often "good enough" for improving general task performance without necessarily compromising model robustness more than standard human data. However, the risk of reinforcing heuristics remains if the synthetic data is biased. Future work should expand this investigation to other tasks beyond NLI, such as social bias or reasoning fallacies, and explore the effects of scaling to larger model sizes (e.g., 70B+ parameters) where the quality and ingrained biases of the generator may differ significantly.

Summary updated

BibTeX
@inproceedings{zhang2025does,
  author = {Lingze Zhang and Ellie Pavlick},
  title = {{Does Training on Synthetic Data Make Models Less Robust?}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2025},
}
Figure