Targeted Data Generation: Finding and Fixing Model Weaknesses

Zexue He, Marco Túlio Ribeiro, Fereshte Khani

Abstract

Even when aggregate accuracy is high, state-of-the-art NLP models often fail systematically on specific subgroups of data, resulting in unfair outcomes and eroding user trust. Additional data collection may not help in addressing these weaknesses, as such challenging subgroups may be unknown to users and underrepresented in existing datasets. We propose Targeted Data Generation (TDG), a framework that automatically identifies challenging subgroups and generates new data for those subgroups using large language models (LLMs) with a human in the loop. TDG estimates the expected benefit and potential harm of data augmentation for each subgroup, selecting the ones most likely to improve within-group performance without hurting overall performance. In our experiments, TDG significantly improves the accuracy on challenging subgroups for state-of-the-art sentiment analysis and natural language inference models, while also improving overall test accuracy.

Introduction

Modern Natural Language Processing (NLP) models often boast impressive aggregate accuracy scores, sometimes exceeding 95%. However, these high-level metrics frequently mask systematic failures on specific "subgroups" or "slices" of data. For instance, a sentiment classifier might perform excellently on standard movie reviews but fail consistently on restaurant critiques or sentences containing multiple conflicting sentiments.

Fixing these "blind spots" manually is a significant challenge. It requires practitioners to first discover unknown weaknesses and then collect or curate specific data to address them—a process akin to a high-stakes game of "Whac-A-Mole." Furthermore, blindly adding data can lead to "interference," where improvements in one niche area inadvertently degrade the model's performance on the broader dataset. To solve this, we introduce Targeted Data Generation (TDG), a scalable framework designed to automatically identify, evaluate, and patch these model weaknesses.

The TDG Framework

The TDG framework operates through a three-stage pipeline: discovery, evaluation, and augmentation.

1. Automatic Subgroup Discovery

Rather than relying on human intuition to find model failures, TDG uses clustering on validation data to surface potential problem areas. The framework evaluates three clustering strategies:

  • Agnostic: Utilizing general-purpose embeddings (e.g., Sentence-BERT).
  • Task-based: Utilizing the target model’s own internal representations from its penultimate layer.
  • Task-based + Label: Clustering based on model representations while ensuring all examples in a cluster share the same ground-truth label.

Research indicates that Task-based clustering is the most effective, as it groups examples based on the model's specific internal logic and linguistic perceptions, effectively highlighting its unique blind spots.

2. Identifying Amenable Clusters

Not every failing subgroup is a candidate for augmentation. Some clusters may be inherently noisy or irrelevant to the primary task. TDG filters clusters using two primary metrics:

  • Generalization in Context (GC): Measures whether providing the model with more data from a specific cluster actually improves its performance on unseen examples from that same cluster.
  • Interference in Context (IC): Measures whether adding data from a specific cluster negatively impacts the model's performance on the original, broader dataset.

The objective is to select "amenable" clusters characterized by High GC and Low IC.

3. Targeted Augmentation with LLMs

Once amenable clusters are identified, TDG leverages Large Language Models (such as GPT-3) to generate new, similar examples. To maintain high data quality, the framework employs a Human-in-the-Loop approach. Human annotators verify the LLM-generated labels and ensure the new data accurately represents the targeted subgroup. The model is then fine-tuned on a curated mix of the original training data and this new, targeted augmentation.

Experimental Results

We evaluated TDG on three benchmark tasks from the GLUE suite: Stanford Sentiment Treebank (SST), Quora Question Pairs (QQP), and Multi-Genre Natural Language Inference (MNLI).

  • SST: TDG improved accuracy on challenging subgroups from 81.45% to 83.60%, while simultaneously increasing overall test accuracy by 0.55%.
  • MNLI: Subgroup accuracy saw a significant jump from 60.57% to 65.03% (a 4.46% absolute gain), with a 0.16% improvement in overall performance.
  • QQP: Interestingly, TDG identified that certain clusters in QQP were not amenable to augmentation due to high label noise (similar sentence pairs with conflicting labels). This demonstrates TDG's utility in diagnosing data quality issues rather than just model weaknesses.

In comparison to standard baselines like Reweighing (oversampling failing examples) and Paraphrasing (using T5 to reword existing data), TDG consistently provided superior gains in both subgroup and aggregate performance.

Key Insights and Discussion

Our findings suggest that fixing niche failures does not have to be a zero-sum game. Improving a specific, failing subgroup—such as sentences with complex sentiment—often provides a marginal but measurable boost to the model's general reasoning capabilities across the entire test set.

However, TDG is best viewed as a "polishing" step for mature, high-performing models. If a model is fundamentally under-trained, standard broad-based data collection remains the priority. Current limitations include the human effort required for label verification (approximately 90 minutes per cluster) and the need for more sophisticated management of overlapping cluster relationships to further minimize interference.

Conclusion

Targeted Data Generation provides a systematic path toward more robust and reliable NLP models. By automating the discovery of blind spots and using LLMs to generate high-quality, targeted training data, TDG allows developers to move beyond aggregate metrics and address the specific failures that erode user trust.

Summary updated

Abstract

Even when aggregate accuracy is high, state-of-the-art NLP models often fail systematically on specific subgroups of data, resulting in unfair outcomes and eroding user trust. Additional data collection may not help in addressing these weaknesses, as such challenging subgroups may be unknown to users, and underrepresented in the existing and new data. We propose Targeted Data Generation (TDG), a framework that automatically identifies challenging subgroups, and generates new data for those subgroups using large language models (LLMs) with a human in the loop. TDG estimates the expected benefit and potential harm of data augmentation for each subgroup, and selects the ones most likely to improve within group performance without hurting overall performance. In our experiments, TDG significantly improves the accuracy on challenging subgroups for state-of-the-art sentiment analysis and natural language inference models, while also improving overall test accuracy.
BibTeX
@inproceedings{he2023targeted,
  author = {Zexue He and Marco Túlio Ribeiro and Fereshte Khani},
  title = {{Targeted Data Generation: Finding and Fixing Model Weaknesses}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
  url = {https://openreview.net/forum?id=3iO17FuhZE},
}
Figure