Targeted Data Generation: Finding and Fixing Model Weaknesses

Zexue He, Marco Túlio Ribeiro, Fereshte Khani

Abstract: Even when aggregate accuracy is high, state-of-the-art NLP models often fail systematically on specific subgroups of data, resulting in unfair outcomes and eroding user trust. Additional data collection may not help in addressing these weaknesses, as such challenging subgroups may be unknown to users, and underrepresented in the existing and new data. We propose Targeted Data Generation (TDG), a framework that automatically identifies challenging subgroups, and generates new data for those subgroups using large language models (LLMs) with a human in the loop. TDG estimates the expected benefit and potential harm of data augmentation for each subgroup, and selects the ones most likely to improve within-group performance without hurting overall performance. In our experiments, TDG significantly improves the accuracy on challenging subgroups for state-of-the-art sentiment analysis and natural language inference models, while also improving overall test accuracy.

Introduction

Modern NLP models often achieve impressive aggregate accuracy scores, yet these figures can mask "hidden stratification"—systematic failures on specific, often under-represented, subgroups of data. For instance, a sentiment analysis model might perform exceptionally well on general movie reviews but fail consistently on reviews mentioning specific genres or technical aspects.

Simply collecting more data is often insufficient to resolve these blind spots because the problematic subgroups are frequently unknown to developers or rare in standard datasets. Furthermore, naively augmenting data for a specific subgroup can lead to "interference," where improvements in one area cause performance drops in others. The challenge lies in automatically identifying these failing subgroups and determining which ones can be effectively "fixed" through targeted data augmentation.

The Targeted Data Generation (TDG) Framework

The TDG framework addresses these challenges through a three-step pipeline designed to identify, evaluate, and augment problematic data slices.

1. Automatic Subgroup Discovery

The first step involves clustering validation data to find cohesive subgroups where the model might be struggling. The framework evaluates three distinct clustering strategies:

  • Agnostic: Utilizing general-purpose embeddings like Sentence-BERT.
  • Task-based: Using the target model’s own internal representations (typically the penultimate layer).
  • Task-based + Label: Clustering based on model representations while ensuring all items within a cluster share the same ground-truth label.
Comparison of clustering strategies: (a) Agnostic, (b) Task-based, and (c) Task-based + Label.

2. Evaluating Amenability

Not every identified cluster is a good candidate for augmentation. TDG filters clusters based on two key metrics to ensure the model benefits from new data without suffering from interference:

  • Generalization in Context (GC): This measures whether training on additional data from a specific cluster actually improves performance on unseen examples from that same cluster.
  • Interference in Context (IC): This measures the potential negative impact of training on a cluster's data on the model's performance across the rest of the dataset.

TDG prioritizes clusters that maximize the value of $GC - IC$, selecting "high-potential" regions for the next phase.

3. Subgroup Augmentation with LLMs

For the selected clusters, TDG employs a Large Language Model (LLM), such as GPT-3, to generate new, synthetically similar examples. To maintain high data quality, a human-in-the-loop provides labels and filters the generated outputs, ensuring the new data is both diverse and relevant to the identified weakness.

The complete TDG pipeline: from identifying challenging clusters to LLM-driven augmentation and testing.

Experimental Results

The authors evaluated TDG on three standard GLUE benchmark tasks: SST (Sentiment Analysis), QQP (Paraphrase Detection), and MNLI (Natural Language Inference). The target models were BERT-base and RoBERTa-large.

Key Performance Gains

TDG demonstrated significant improvements over standard baselines like Reweighing (GDRO) and simple Paraphrasing:

  • SST: Average in-cluster accuracy rose from 81.45% to 83.60%, with a 0.55% boost in overall devtest accuracy.
  • MNLI: Average in-cluster accuracy improved from 60.57% to 65.03%, with a 0.16% overall accuracy increase.

Insights on Model Weaknesses

  • Model-Aware Clustering: Task-based clustering proved superior to agnostic embeddings, suggesting that a model's specific failure modes are often encoded within its own internal representations.
  • The Role of Interference: In datasets like QQP, which can contain high label noise, the IC metric successfully flagged clusters where augmentation would have been counterproductive, preventing model degradation.
  • Surgical Precision: Rather than broad data augmentation, using LLMs as a "surgical" tool to fill specific gaps proved highly effective. Interestingly, fixing these minority subgroups often led to a general robustness boost across the entire test set.

Conclusion and Future Work

TDG provides a systematic way to move beyond aggregate metrics and address the specific weaknesses of NLP models. By combining automatic discovery with LLM-based generation and human oversight, the framework improves both subgroup fairness and general model performance.

Future research may explore inter-cluster relationships to understand how augmenting one group affects another, and investigate more automated labeling techniques to further reduce the human effort required in the pipeline.

Summary updated

Abstract

Even when aggregate accuracy is high, state-of-the-art NLP models often fail systematically on specific subgroups of data, resulting in unfair outcomes and eroding user trust. Additional data collection may not help in addressing these weaknesses, as such challenging subgroups may be unknown to users, and underrepresented in the existing and new data. We propose Targeted Data Generation (TDG), a framework that automatically identifies challenging subgroups, and generates new data for those subgroups using large language models (LLMs) with a human in the loop. TDG estimates the expected benefit and potential harm of data augmentation for each subgroup, and selects the ones most likely to improve within-group performance without hurting overall performance. In our experiments, TDG significantly improves the accuracy on challenging subgroups for state-of-the-art sentiment analysis and natural language inference models, while also improving overall test accuracy.
BibTeX
@inproceedings{he2023targeted,
  author = {Zexue He and Marco Túlio Ribeiro and Fereshte Khani},
  title = {{Targeted Data Generation: Finding and Fixing Model Weaknesses}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
  url = {https://openreview.net/forum?id=3iO17FuhZE},
}
Figure