Quality Filtering for High Fidelity Synthetic Data

This direction is split into other directions: selective data augmentation. Its last version stays readable and citable here.

Version 1 of 1628 papers791 conceptsfrozen 2026-10-04

In the era of massive neural networks, the bottleneck for performance has shifted from model architecture to data availability. While generative models like GPT-4 or Diffusion models can produce an infinite stream of synthetic training data, this "abundance" comes with a catch: raw synthetic data is often riddled with hallucinations, logical inconsistencies, and statistical noise.

The emerging consensus across research—from protein prediction to document analysis—is that quality filtering is more critical than data volume. By placing a "critic" or "filter" between the generator and the final learner, researchers can distill noisy synthetic outputs into high-fidelity training signals that sometimes outperform human-annotated data.

The Generator-Filter-Learner Pipeline

The core mechanism shared across these works is a three-stage pipeline often described as "machine-to-corpus-to-machine." Instead of training a model directly on the output of a generator, a secondary process evaluates each synthetic sample for "plausibility" or "utility."

Chandra Bhagavatula and colleagues demonstrated this in Symbolic Knowledge Distillation. They used GPT-3 to generate a massive commonsense knowledge graph but found the raw output too noisy for training. By applying a "critic" model to prune illogical examples, they trained a student model 100x smaller than GPT-3 that actually outperformed its teacher. The filter acts as a "sanity check," ensuring the student learns from a clean signal rather than the teacher's hallucinations.

Filtering by Model Confidence

One of the most effective ways to filter data is to use a pre-trained classifier to "grade" the synthetic samples. This is the intuition behind Supervised Over-Sampling (SOS), introduced by Jun Hu and Hong‐Bin Shen in A New Supervised Over-Sampling Algorithm.

In their work on protein binding sites, they used a pre-trained classifier to evaluate synthetic data points. If the classifier was too confident, the sample was redundant; if it was too uncertain, the sample was likely noise. By targeting an intermediate confidence range, they forced the final model to learn from "challenging yet valid" instances. Similarly, Vincent Claveau found in Generating Artificial Texts that using a classifier to remove generated texts that didn't align with their intended labels significantly boosted the performance of traditional models.

Enforcing Physical and Logical Consistency

In domains like computer vision and medical imaging, filtering often takes the form of "consistency checks." Synthetic data can easily produce "impossible" scenarios—like a rust fault appearing in mid-air or a medical scan with anatomically impossible organs.

  • Spatial Consistency: Xuan Yi and colleagues in Appearance Fault Diagnosis of a Transformer used a secondary classification layer to discard synthetic images where textures were placed in semantically incorrect locations.
  • Task-Based Consistency: In the realm of Question Generation, Alireza Mohammadshahi proposed RQUGE, a metric that evaluates a generated question by checking if a separate QA model can actually answer it correctly. If the question is unanswerable, it is filtered out of the training set.
  • Multimodal Alignment: For complex tasks like Composed Person Retrieval, Delong Liu used multimodal models to ensure that synthetic text descriptions actually matched the visual changes in the synthetic images.

Learning Dynamics as a Filter

A more recent and sophisticated approach involves looking at how a model behaves during training to identify bad synthetic data. Nabeel Seedat and Mihaela van der Schaar introduced CLLM (Curated LLM), which filters synthetic tabular data based on "learning dynamics."

In Curated LLM, they argue that samples which cause high uncertainty or erratic prediction patterns in a proxy model are likely noise. By discarding these "unstable" samples, they achieved state-of-the-art results in ultra-low-data regimes. This suggests that the process of learning can itself be a filter, identifying data points that contradict the underlying patterns of the real-world distribution.

Why It Works: The Signal-to-Noise Ratio

The intuition behind all these methods is that deep learning models are "picky eaters." As noted by Christian Falkenberg in Mr-Fosdick at SemEval-2023, high-capacity models like Transformers can actually perform worse when fed low-quality synthetic data because they are prone to overfit on the artifacts and errors of the generator.

Filtering acts as a denoising step that restores the signal-to-noise ratio. Whether it is Bin Wang discarding the majority of generated dialogues to ensure uniqueness in InstructDS, or Christian Klötergens using rejection sampling to exclude unstable trajectories in Physiome-ODE, the goal is the same: prioritize the fidelity of the training distribution over its scale.

Papers in this version

Showing 60 of 628 papers.

History & citationVersion 1 · 2026-10-04

Split into other directions · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_quality_filtering_for_high_fidelity_synt_5c27744e,
  title = {Quality Filtering for High Fidelity Synthetic Data},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/quality-filtering-for-high-fidelity-synthetic-data/rdr_5c27744ee9b141f3a555ea765f6b9ca4},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_quality_filtering_for_high_fidelity_synt_5c27744e_v1,
  title = {Quality Filtering for High Fidelity Synthetic Data},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/quality-filtering-for-high-fidelity-synthetic-data/rdr_5c27744ee9b141f3a555ea765f6b9ca4/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}