When Execution Filters Learn Who Speaks "Proper" English

Code assistants are trained more and more on synthetic data: a large model writes a programming instruction and a solution, the solution is run against unit tests, and only pairs whose code passes are kept. This execution check is treated as neutral, since a test either passes or fails. We argue it is not neutral when the instructions vary in how they are written. Suppose a task is phrased in African American English, Nigerian English, learner English, or terse chat style. If the generator misreads that phrasing, its solution fails, and the filter discards the pair. The filter then removes non-standard phrasings at a higher rate than standard ones. The surviving data teaches the next model that working code follows from textbook English. If that model becomes the next generator, the gap can widen with each round.

This note proposes to measure that effect and to remove it with one structural change, which we call paired verification. Under paired verification, a solution is checked by execution against a canonical, standard-register statement of the task. The verified solution is then attached to several register rewrites of the same task. Whether a pair is accepted then depends on whether the code is correct, not on whether the generator understood the dialect. The setting also offers a way out of a long-standing problem in style-transfer evaluation. In code, "did the rewrite preserve the meaning?" can be partly anchored by fixed examples, fixed constraints, and hidden tests, instead of by overlap metrics that are known to fail.

Core Claim

Execution-based filtering of synthetic code-instruction data acts as a register-biased selector. Across repeated rounds of training on its own filtered output, this selector amplifies a functional-correctness penalty against users who write in non-standard varieties of English. Separating verification (done on a canonical statement) from instruction phrasing (attached afterward) should close most of that penalty at no cost to standard-English performance. It should also do better than the obvious fix of rewriting user prompts into standard English at inference time.

The two quantities we track are the register gap and the survival ratio. For a model, a register $r$, and a held-out problem set, the register gap is

$$\Delta_r = \text{pass@1}(\text{standard}) - \text{pass@1}(r),$$

where pass@1 is the fraction of problems solved on the first sample. For a filter applied to synthetic data, the survival ratio is

$$s_r = \frac{\text{acceptance rate of pairs written in } r}{\text{acceptance rate of standard-register pairs}}.$$

The amplification hypothesis predicts three things. First, $s_r < 1$ under ordinary execution filtering. Second, $\Delta_r$ grows across self-training rounds. Third, under paired verification $s_r$ equals 1 by construction and $\Delta_r$ shrinks.

Where This Sits

The execution side supplies the method and also the blind spot. APPS, XCodeEval, and LiveCodeBench built test-based benchmarks whose problem statements are scraped from contests in standard English. Execution is therefore well defined, but the phrasing of the specification never varies. LiveCodeBench's release-date tagging lets us evaluate on problems published after a model's training cutoff, and we reuse that design so that a measured register gap cannot be blamed on memorized problems.

Execution as a filter for synthetic data comes from TableLLM and Scaling Code-Assisted Chain-of-Thoughts. TableLLM admits a synthetic sample only when a reasoning path and an executed Pandas program agree. The second paper uses code execution to stop arithmetic errors from spreading through reasoning traces. Ensemble-Instruct instead keeps outputs on which a committee of models agrees. All three judge only whether the output is correct. None asks which inputs the filter systematically throws away. We reuse their machinery and study its selection effect.

The bias side motivates the work. Fairness Feedback Loops showed that repeatedly training on a model's own synthetic output can completely erase a minority group within a few generations. That study used image and classification data, and it proposed stratified resampling as a repair. AttrPrompt found that naive synthetic text was 68% about North America and 0.69% about Africa, and it repaired the skew by conditioning generation on attributes. Linguistic Bias in ChatGPT and Uncovering Stereotypes in LLMs report that model behavior degrades or becomes stereotyped under dialectal variation and more complex prompts. We carry the feedback-loop question into code. In code, the group that gets erased is a way of writing, and the eraser is a unit test rather than a classifier.

The closest prior work is Bias Testing and Mitigation in LLM-based Code Generation. It executes generated decision-making code under inputs that differ only in protected attributes, and reports that GPT-4-turbo produces age-biased code in 52.10% of tasks. In that work, the bias lives in the program's logic about people. Here, the bias lives in which users get working programs, and in how a data pipeline compounds that over rounds. We use their test pool only as a secondary readout. That paper also does not study filtering or recursive training.

The style literature constrains how we validate rewrites. Mind the Style Gap showed that BLEU and Meteor correlations with human judgments fall to zero or below once stylistic shifts are large. Expertise Style Transfer showed that style classifiers can be gamed with surface prefixes. We therefore do not use any overlap metric or style classifier to certify that a dialect rewrite kept the task's meaning. Beyond Functional Correctness and Lightning OPD 2.0 study style as a property of the output: code conventions, and the phrasing of reasoning traces. We study style as a property of the input and of who wrote it.

Research Program

The first component is a validated set of register rewrites. We take LiveCodeBench problems released after the cutoffs of the models under test. Each narrative statement is rewritten into six registers: African American English, Indian English, Nigerian English, learner English with typical second-language patterns, terse chat style, and a standard-English paraphrase. The standard paraphrase is a control. It separates "any rewording hurts" from "non-standard varieties hurt." Sample input/output blocks, numeric constraints, and format specifications are locked spans, and we check by exact string match that they are unchanged.

Meaning preservation of the free text is the hard part, and model judgment cannot settle it. A model that misreads the dialect would make the same mistake when judging the rewrite. Instead, fluent speakers of each variety rate a stratified subsample on two questions: whether the rewrite says the same thing, and whether it reads as authentic rather than caricatured. Rewrites failing either check are discarded. All headline gap numbers are reported only on the human-validated subset.

The second component is the filtering study. We start from the 5,000 APPS training problems and their tests. A generator model, an open code model, produces instruction-solution pairs in each register. We compare five ways of curating that data. Raw keeps every pair. Standard execution filtering generates the solution from the instruction as written and keeps the pair if the solution passes the tests. Committee consensus follows Ensemble-Instruct but agrees on test outcomes instead of ROUGE-L. Stratified reweighting applies STAR-style repair from Fairness Feedback Loops after execution filtering. Paired verification is our method. Each curation produces a training set for a 7B student. The student then becomes the generator for the next round, for four rounds. At every round we record $s_r$ and $\Delta_r$.

The third component is a secondary bias readout. We rewrite the decision-task prompts from Wang et al.'s test pool into the same registers and ask whether the Code Bias Score changes with the register of the request. A pipeline that teaches models to take non-standard writers less seriously might also change what code they write about people.

Baselines

The naive reference point is the base generator with no synthetic fine-tuning, evaluated on the same validated rewrites. It shows the register gap that pretraining alone produces, and every curated student is compared against it. To our knowledge from the file, no reported result found on Lacuna for pass@1 under dialect- or register-rewritten code specifications. We therefore measure this reference ourselves. The nearest reported figure for the benchmark is LiveCodeBench's GPT-4-Turbo rising from 24.5% to 36.9% pass rate with self-repair (LiveCodeBench), which applies only to standard-English statements.

The strongest existing curation method is standard execution filtering of the kind used by TableLLM. Its cross-way validation is reported to bring an 8B model to 86.74% average accuracy on office benchmarks and 81.05% on Spider, against 69.53% for GPT-4o. It is the strongest competitor because it is what practitioners actually run, and it optimizes the correctness that paired verification must not lose. Committee consensus following Ensemble-Instruct is a second strong competitor. That paper reports 46.8 ROUGE-L on SuperNI, against 39.9 for Self-Instruct data, though that number comes from a non-code setting. Stratified reweighting following Fairness Feedback Loops is the strongest fairness-aware competitor.

For a fair comparison, all methods start from the same APPS problem pool, the same generator, and the same budget of sampled solutions per problem. Every training set is truncated to the same size, so no method wins by having more data. All methods get the same hyperparameter search budget on a held-out split of standard-English APPS problems. Paired verification beats a baseline if it reduces the mean $\Delta_r$ across non-standard registers by more than the spread across three random seeds, while keeping standard-register pass@1 on post-cutoff LiveCodeBench within one point of that baseline.

Two ablations isolate the key ingredient. The first is paired verification with only standard-English instructions attached, which tests whether the gain comes from verified data alone or from exposure to register variation. The second is register-diverse data with no filtering, which tests whether the gain comes from diversity alone or from diversity combined with correctness.

Why not just…?

Why not just rewrite user prompts into standard English at inference time? This is the most serious threat, because it needs no retraining. It has three weaknesses. The normalizer is itself a model reading the dialect, so it can lose meaning in exactly the cases where meaning is hardest to recover. It adds latency and cost on every call. And it leaves the underlying data loop untouched. We test it directly. A strong model normalizes each validated rewrite, and the base and curated students are evaluated on the normalized versions. If normalization closes the gap as well as paired verification, we will report that the cheap fix suffices for single-round use. The amplification result across self-training rounds would still stand as a warning about the data pipeline.

Why not just add unverified paraphrases to the training data? Unfiltered register augmentation is cheaper, but it reintroduces the incorrect solutions that execution filtering exists to remove. The ablation with register-diverse, unfiltered data measures this tradeoff. If that ablation matched paired verification on both register gap and standard-English pass@1, the filtering half of our claim would fail.

Why not just reweight the surviving data by register, as STAR or AttrPrompt would? Reweighting can restore the share of each register, but only from the pairs that survived filtering. Those survivors are biased toward the dialect phrasings the generator happened to read correctly, which are likely the easiest and most standard-like ones. We expect reweighting to fix the proportions without fixing the content. The comparison should show this as $\Delta_r$ staying high even when register shares are equalized. If it does not, reweighting is the simpler answer, and we will say so.

Decisive Checks

Four results decide the proposal. First, on human-validated rewrites, the gap for non-standard registers must exceed the gap for the standard-paraphrase control. If it does not, the effect is "rewording hurts," not a register penalty, and the project becomes a robustness study. Second, standard execution filtering must show $s_r$ reliably below 1. If it does not, filtering is not the mechanism. Third, $\Delta_r$ must grow across the four self-training rounds under standard filtering and stay flat or shrink under paired verification. This is the code analogue of the erasure reported in Fairness Feedback Loops. Fourth, paired verification must beat both the inference-time normalizer and the reweighting baseline under the equal-budget protocol above.

Secondary readouts are the Code Bias Score under register-rewritten prompts, and the share of style defects from the Beyond Functional Correctness taxonomy. The latter checks whether the fix trades functional fairness for worse code conventions.