Diagnosing Rejected Synthetic Samples Before Repairing Them

Pipelines that build synthetic reasoning data with a teacher model often check each sample twice. The model solves a problem once in natural language and once as an executable program, and the sample is kept only if the two answers agree. This check is behind strong math and table-reasoning datasets, but it is a blunt gate: every disagreement is thrown away. The natural fix is to run a debugging loop on rejected samples, feeding the traceback back to the model until the two answers agree. We argue that this fix is dangerous when applied blindly. A loop that rewards agreement can manufacture agreement. It can patch the code until it prints whatever the text said, or force a confident answer onto a problem that admits two readings. The result recovers volume and corrupts the data.

Our proposal places a diagnostic step between rejection and repair. For each sample that fails the agreement check, the pipeline decides why it failed, using execution rather than the model's own verbal self-assessment. It distinguishes three causes. In an execution fault, the code crashed or has a local bug, and the sample is worth repairing. In an ambiguity, the problem statement supports two coherent answers; such a sample should be rewritten or dropped, never "fixed." In a knowledge gap, no solver produces a stable answer, and the sample should be dropped and logged as a hard region. Only the first class enters the repair loop. Every repair must then pass a check that it was not achieved by copying the target answer. The claim is that cause-aware routing recovers most of the useful data that blind repair recovers while keeping the precision of the admitted set at or above the discard-only baseline. Under equal generation compute, this should also beat simply sampling more.

Where this sits in prior work

Three lines of work supply the pieces. The first supplies the problem. Caco converts math solutions into Python templates, executes them, back-translates them into problems and chain-of-thought (step-by-step written reasoning), and admits a pair only when text and execution agree. TableLLM applies the same "cross-way validation" with Pandas code for spreadsheet tasks. Both discard disagreements outright, and neither records why a sample failed. Case2Code uses an interpreter as the source of ground truth but starts from human code that already runs. These papers motivate our work: they show execution-anchored filtering produces strong training data, and they leave the rejected pool unexamined.

The second line supplies evidence that repair works, and also a warning. CodeTransOcean reports that feeding compiler errors back to ChatGPT raises its debugging success rate from 48.57% to 52.57% over three rounds, with most of the gain in round one. That suggests some failed samples are cheaply recoverable, and that gains plateau quickly. ICER adds a constraint on how feedback should be shaped. Asking models to state their code's Big-O complexity hurts refinement, and frontier models label complexity correctly only about 60% of the time. The same benchmark paper finds plain chain-of-thought hurts translation. We take this as a design constraint: the diagnosis should come from execution probes the pipeline runs, not from asking the solving model to explain itself. SlopCodeBench shows a related hazard. Agents iterating on their own code keep passing tests while quality erodes, so "tests pass" is a weak acceptance signal for iterative edits.

The third line supplies the diagnostic method. Rejection-transparency work argues that a refusal is only useful if it says what kind of uncertainty caused it. Brinkrolf et al. separate borderline rejects, which fall between two well-understood classes, from outlier rejects, which fall far from all of them. They validate the split by controlled perturbation: interpolating between prototypes produces borderline rejects, and adding noise produces outliers. L2loRe rejects when the nearest counterfactual is close. Artelt et al.'s semifactual explanations map which input changes leave a rejection unchanged. We transfer these ideas from feature space to problem-parameter space. Caco-style problems come from code templates, so we can perturb a problem's numbers cheaply and watch whether the disagreement flips, persists, or splits into stable modes. This is the counterfactual-and-semifactual analysis of a reject, run on a data sample instead of a classifier input. The closest work joining transparency and synthetic data, TRU, trains models on teacher-written refusal rationales. Its rationales are produced one-shot and are never checked by execution. That is the weakness we avoid.

Core Claim

When a two-way consistency filter rejects a synthetic sample, three things can be read from the rejected sample with execution alone, and they determine whether repair helps or harms. The first is how the disagreement behaves under small perturbations of the problem's parameters. The second is whether independent re-solves cluster into one answer, several stable answers, or none. The third is whether the code ran at all. Routing on this diagnosis, combined with a repair check that requires agreement to survive perturbation, should yield a training set that is both larger than discard-only and cleaner than blind repair.

Research Program

The pipeline starts from a Caco-style generator: seed math solutions turned into Python templates, instantiated, back-translated into a problem and a chain-of-thought, and checked for agreement. For every disagreement it runs three inexpensive probes. The first is an execution probe: did the program crash, time out, or return a malformed value? The second is a re-solve probe. A small number of independent solvers, possibly from different model families as in Ensemble-Instruct, each produce a fresh code-and-text pair, and we record whether the self-consistent pairs converge on one answer, split into two or more stable answers, or scatter. The third is a perturbation probe. The template's numeric parameters are resampled several times, and we record whether the original disagreement reappears.

The routing rule follows the uncertainty taxonomy. A sample whose code failed, or whose disagreement vanishes under most perturbations, is treated as an execution fault or local edge-case bug. It enters a repair loop of at most three rounds, the point where CodeTransOcean's gains stop. A sample whose re-solves split into two or more stable, internally consistent answers is treated as ambiguous. The double cone in Brinkrolf et al. is the analogy: a sample sitting between two coherent modes. Such a sample is sent to a rewrite step that adds a disambiguating clause and must pass the full check again, or else it is dropped. A sample whose re-solves scatter, or whose disagreement persists across perturbations with no stable mode, is treated as a knowledge gap and dropped. Its template is logged as a hard region.

Repair acceptance is where blind repair goes wrong, so we make it strict. A repaired sample is admitted only if the repaired code agrees with freshly generated chain-of-thought on several perturbed instances of the same template, not just on the original. The repair diff also must not introduce constants equal to the target answer. This perturbation check is the ingredient we expect to matter most, and it gets its own ablation.

The pipeline also produces a per-sample rejection record: the cause, the probe evidence, and the action taken. We release it as a diagnostic of where the generator is weak. We also test whether the hard-region log is useful for targeted regeneration, but that is secondary to the main claim.

Baselines

The naive reference point is the discard-only filter as described for Caco: execute, check dual agreement, drop everything else. We reimplement it on the same seeds, generator, and student. For context, Caco reports that Qwen2.5-7B trained on Caco-1.3M reaches 82.4% on MATH and 92.6% on GSM8K, and that LLaMA3-8B rises from 14.5% to 34.1% on OlympiadBench. Our runs will be smaller, so these figures serve as an external reference, not a target we claim to match at reduced scale.

The second baseline is resample-to-budget. It also discards failures, but it spends the compute our probes and repairs consume on generating additional fresh samples, so total generation tokens are identical. This is the strongest competitor on our specific question. If fresh samples are as cheap and as good as recovered ones, diagnosis is pointless. Beating it means higher downstream accuracy at equal tokens, not just a larger dataset.

The third baseline is blind repair: every rejected sample gets up to three rounds of traceback-and-disagreement feedback and is admitted once the two answers agree. This is the obvious combination of the two directions. For repair inside a synthetic-data filter, no reported result was found on Lacuna. The closest number is CodeTransOcean's three-round gain from 48.57% to 52.57% in a different task.

The fourth baseline replaces our execution probes with a verbal diagnosis. A strong model is shown the rejected sample and asked to label the cause, and routing follows its label. All baselines share the same generator, student architecture, training steps, and hyperparameter search budget, and we report three seeds. Two ablations isolate the key ingredients: routed repair without the perturbation acceptance check, and routing with only the execution probe (no re-solves or perturbations).

We count the proposal as successful if three conditions hold across seeds. Routed repair must beat both discard-only and resample-to-budget on MATH and OlympiadBench at matched tokens. Blind repair must show lower admitted-set precision than routed repair. And removing the perturbation check must measurably lower precision.

Why not just…?

Why not just discard failures and generate more? This is the most serious threat, and the resample-to-budget baseline exists to settle it. The case against it is indirect but real. Discarding removes exactly the samples near the solver's error boundary. WizardCoder's results suggest harder, more complex instructions carry disproportionate training value (WizardCoder). Ensemble-Instruct shows curated data can match far larger uncurated volume: 30k filtered samples matched 82k GPT-3 samples. Neither result proves that recovered samples beat fresh ones. If resample-to-budget ties routed repair, we will report that repair adds nothing beyond volume. The remaining contribution would then be the rejection record as a diagnostic of the generator.

Why not just run a debugging loop on every rejected sample? Because the loop's success signal is agreement, and agreement can be gamed. The code can be edited to emit the text's answer, and an ambiguous problem can be forced to one reading. SlopCodeBench shows passing tests coexisting with degrading code, which is the same failure of a weak acceptance signal (SlopCodeBench). The blind-repair baseline, scored by admitted-set precision on audited and fault-injected samples, will show directly whether this corruption occurs. If blind repair turns out to be as precise as routed repair, our core claim fails.

Why not just ask a strong model why the sample failed? ICER finds that frontier models label their own code's complexity only about 60% of the time, and that requesting such labels hurts refinement (ICER). The abstention literature shows that prompt-only abstention can fail almost completely: an abstention F1 of 2.18 versus 55.47 for calibrated contrastive decoding on Natural Questions (CDA). These results suggest verbal self-diagnosis is unreliable. Still, a different, stronger model judging someone else's sample might do better, so the verbal-diagnosis baseline is run head to head and scored on the same labeled faults.

Proposed Experiments

The main domain is math, following Caco's setup. Students are Qwen2.5-7B and LLaMA3-8B, evaluated on MATH, GSM8K, and OlympiadBench-Math. A second domain checks transfer: TableLLM-style cross-way validation on spreadsheet tasks, evaluated on WikiTQ and Spider. There, TableLLM reports 89.10% on WikiTQ and 81.05% on Spider for its 8B model.

The first decisive check validates the diagnoses themselves, in the spirit of Brinkrolf et al.'s perturbation tests. We take clean, agreeing samples and inject known faults: syntax or off-by-one bugs into the code, arithmetic slips into the text, and deliberate dual readings into the problem statement. We then measure how accurately each diagnosis method recovers the injected cause. A random sample of naturally rejected items is also audited by hand.

The second check measures admitted-set precision. For each pipeline, a strong model checks a held-out sample of admitted items, with a human spot-audit, to estimate how many admitted samples are actually wrong.

The third and decisive check compares downstream accuracy under matched generation tokens across all baselines and ablations. We also report the yield curve: how many samples each method recovers per million extra tokens, and at what precision.

The proposal stands if three things hold. The fault-injection study must show that execution probes beat verbal diagnosis at identifying causes. Blind repair must admit measurably more wrong samples than routed repair. And routed repair must beat resample-to-budget downstream. If only the first two hold, we have shown how to repair safely, but not that repair is worth its compute.