Native-Speaker-Checkable Specifications for Low-Resource Code Generation

When someone describes a programming task in Swahili, Zulu or Nyanja instead of English, open-source code models often fail completely. The failures are not subtle. Models collapse to near-zero pass rates and sometimes translate reserved words like if and return into the prompt's language, which breaks the code outright. The obvious fix is training data: pairs of low-resource prompts and correct reasoning. But checking such data normally needs someone who both speaks the language and can program, and in most of the world's languages that person is scarce. We propose to split the check in two. A model first restates the task as a short structured specification, written in the user's language with code-level terms left in English. The specification states the inputs, the outputs, the branch and loop conditions, and the edge cases. A native speaker who cannot program checks only whether this restatement matches what the prompt asked. Running the unit tests checks whether the code matches the specification.

The second part of the idea decides where the scarce human effort goes. For each training problem we derive two specifications, one from the native-language prompt and one from an English translation of it. Humans review only the steps where the two disagree. The corrected specifications, each paired with test-verified code, become fine-tuning data that teaches an open model to write its own specification before coding. The claim is concrete. Per annotator-hour, this data should close more of the low-resource gap than two cheaper options: fully automated synthesis, and simply translating prompts into English.

Core Claim

Our hypothesis is that much of the low-resource failure in code generation is a comprehension failure, not a coding failure. The model maps the prompt to the wrong problem, and it then writes code that is internally coherent but solves something else. If so, three things follow. A structured specification is the right unit of human review, because comprehension errors become visible there in a form a non-programmer can judge. Disagreement between the native-derived and English-derived specification is a cheap signal that tells us where the comprehension errors are. And the comprehension fixes that execution-filtered distillation cannot supply will show up as gains on held-out low-resource languages.

If the hypothesis is false, our first diagnostic will show it. In that case failures persist even when the model is handed the original English prompt, and the bottleneck is code synthesis rather than understanding.

What Prior Work Establishes and Leaves Open

Two benchmarks supply the evidence for the gap, and both are confined to standalone functions with direct prompting. mHumanEval covers 204 natural languages in six resource classes. It reports that Claude 3.5 and GPT-4o stay above 0.70 Pass@1 in Python across most classes, while open-source models such as WizardCoder fall to near 0% on low-resource languages. It also documents the keyword-translation bug. HumanEval-XL reports GPT-4 Python Pass@1 of about 78.5% on high-resource languages and about 77.6% on lower-resource ones.

This near-parity constrains the proposal. Frontier models barely need help on standalone tasks, so our target is open models, the rarest language classes, and the use of frontier models as teachers. Neither benchmark tests any intermediate reasoning.

Our method borrows from two existing techniques. Structured Chain-of-Thought Prompting for Code Generation has the model write sequence, branch and loop structure plus input–output constraints before coding. Chain-of-thought here means emitting intermediate reasoning steps before the answer. In English it reports relative Pass@1 gains over ordinary chain-of-thought of up to 13.79% on HumanEval, 12.31% on MBPP and 6.63% on MBCPP. We reuse its scaffold but change its job. The scaffold becomes a bilingual artifact that a human checks for requirement fidelity, not only a prompt that helps the model.

Human-in-the-Loop through Chain-of-Thought supplies the idea of targeted repair. It has humans fix only the reasoning step that sampled answers disagree on. Correcting one step took 21 seconds against 49 seconds for solving the problem from scratch on AddSub, and accuracy on SVAMP rose from 68.00% to 80.10%. That work is English-only arithmetic, uses humans at inference time, and assumes the corrector can solve the problem. We move the human to training-data construction. We replace within-language sampling disagreement with cross-lingual disagreement, and we deliberately use correctors who cannot solve the coding problem.

Three cross-lingual reasoning methods push in the opposite direction: they route reasoning through an English pivot. Cross-Lingual-Thought prompting translates the query into English, reasons there, and formats the answer. Cross-lingual Prompting separates an English alignment step from solving and votes over paths sampled in several languages. XCOT fine-tunes on machine-translated GSM8K rationales with no human validation and reports a 15% average gain on MGSM, the multilingual grade-school math benchmark.

All three work on natural-language math or classification, not code. Disentangling Language Understanding and Reasoning Structures complicates the pivot story. It finds language-specific reasoning neurons whose deactivation costs up to 27.4% on Japanese MGSM, which suggests native-language reasoning is not simply a worse copy of English reasoning. We test this tension directly by varying the language the specification is written in.

On the code-quality side, two papers give us the error vocabulary we use to label failures. Where Do LLMs Still Struggle? names "Wrong Problem Mapping" as a recurring failure pattern. LLM Hallucinations in Practical Code Generation attributes 43.53% of hallucinations to Task Requirement Conflicts. Both studies were run with English prompts only.

Research Program

Diagnostic: is the failure comprehension? We first determine how much of the low-resource gap is comprehension. We take open code models, for example DeepSeek-Coder, on mHumanEval prompts in roughly eight Class 0–2 languages where we can recruit native speakers. Nyanja, Zulu, Swahili and Quechua are among the candidates. For each failure we rerun the same problem under three conditions:

  • the original English prompt;
  • a high-quality machine translation into English;
  • a teacher-written English specification.

We then label the residual failures with the two taxonomies above. The size of the "English prompt fixes it" fraction tells us how much room a comprehension-focused method has.

Data construction. Training problems come from MBPP-style tasks, translated into the target languages with native-speaker spot checks so that we never train on the HumanEval problems we evaluate on. A frontier teacher writes a specification from the native prompt, writes a second specification from an English translation, and writes code for each. Steps where the two specifications disagree, about inputs, outputs, conditions or edge cases, are flagged. A native speaker sees the original prompt and a specification in their language, with code identifiers kept in English, and either accepts each flagged step or edits it in one sentence. The teacher then regenerates code from the corrected specification, and only code that passes the tests is kept. Annotator time is logged per item.

Training. An open student model is fine-tuned to emit the specification and then the code when given a low-resource prompt. At inference time no human is involved.

Specification language. We vary the language of the specification across three settings: native language, English, and native language with English glosses. This tests whether the English-pivot or the native-circuit account better predicts code outcomes.

Extension to repository code. As a secondary probe of whether context-dependent code makes things worse, we translate a subset of CoderEval docstrings into three languages. CoderEval reports ChatGPT Pass@10 falling from about 47.6% on standalone functions to about 23.3% on non-standalone ones in English. On this subset we test whether the specification step still helps when the task depends on project context.

Baselines

Do nothing. The naive reference point is the open student prompted directly in the low-resource language. mHumanEval reports near-0% Pass@1 for open models on low-resource classes, so this sets the floor. The frontier teacher prompted directly sets a practical ceiling, above 0.70 Pass@1 per mHumanEval.

Translate-then-code. This is the strongest training-free competitor. We translate the prompt to English with both NLLB and GPT-4o and give the result to the same student. It is strong because the output is code, not natural language, so a decent translation may be all the comprehension needed. Cross-lingual Prompting reports that its alignment prompt beats Translate-En on reasoning tasks. However, no reported result was found on Lacuna for translate-then-code on mHumanEval or HumanEval-XL, so we must measure it ourselves.

Execution-filtered distillation. This is the strongest existing training-data method in spirit. The teacher writes code directly for low-resource training prompts, only test-passing outputs are kept, and the student is fine-tuned on the survivors. This mirrors the fully automated XCOT recipe, adapted with tests as the filter. The closest published number is XCOT's 15% average MGSM improvement over MathOctopus, which is a math result and not directly comparable. No code result was found on Lacuna.

Automated specification synthesis. The same specification pipeline is run with humans replaced by a teacher acting as judge plus round-trip back-translation filtering, using the BERTScore 0.95 threshold from HumanEval-XL.

Fairness and ablations. Every training arm uses the same problems, the same student, the same fine-tuning compute and tokens, and the same hyperparameter search budget. The human arms are compared at matched annotator-hours. Two ablations isolate the key ingredient. The first gives humans randomly chosen specification steps at the same budget, which isolates the disagreement trigger. The second has programmer-annotators review code instead of specifications at the same budget, which isolates the claim that non-programmer review of specifications is the efficient unit.

What counts as winning. The proposal wins if disagreement-flagged human correction beats both execution-filtered distillation and automated specification synthesis in held-out-language Pass@1, at equal compute and per annotator-hour. It must also beat random flagging, and the gain must concentrate on failures labeled Wrong Problem Mapping.

Why not just…?

Why not just translate the prompt into English and code from that? This is the most serious threat, and we treat it as a possible outcome, not something to argue away. If the diagnostic shows the English prompt recovers most failures and good translation into English recovers nearly as many, then the right conclusion is that machine translation suffices wherever it is good. Our method would then matter only where translation is poor, which is likely in Class 0 languages. We report the gap language by language against the translation quality of each language, so the reader can see exactly where translation stops being enough.

Why not just use unit tests as the verifier and skip humans entirely? Tests confirm that the code solves the problem the tests encode. They cannot tell us whether the student learned to read the low-resource prompt or learned to recognize a familiar problem from surface cues. They also cannot help on new problems that have no tests. LLM Hallucinations in Practical Code Generation finds requirement conflicts to be the largest error category even with English prompts. The execution-filtered baseline settles this question. If it matches our method on held-out languages and on paraphrased prompts that test comprehension, the human step is unnecessary and we will say so.

Why not just prompt with plain or structured chain-of-thought in the target language? Structured Chain-of-Thought prompting helps in English. But a model that misreads the prompt will write a fluent specification of the wrong task, and nothing in prompting corrects that. We include structured prompting without fine-tuning as an arm. That separates the gain from the format itself from the gain from the corrected data.

Why not just use a bigger model? HumanEval-XL's near-parity for GPT-4 shows that scale largely solves standalone tasks for frontier models. That is why the frontier model serves as our teacher, not as the target. The question we address is how to transfer that ability into open models that can be deployed cheaply, where the mHumanEval collapse is documented.

Decisive Checks

We evaluate on mHumanEval and HumanEval-XL in Python. We report Pass@1, the syntax-error rate, the rate of translated keywords, and the share of failures labeled Wrong Problem Mapping, along with annotator seconds per accepted item. Pass@1 is measured on mHumanEval-Expert prompts, which are human-translated, to rule out artifacts of machine-translated prompts. We also report results on two or three languages held out from training.

Three outcomes would falsify the proposal. The first is a diagnostic showing that fewer than a quarter of low-resource failures are fixed by the English prompt, which would mean the bottleneck is not comprehension. The second is that disagreement-flagged correction does no better than random flagging at equal budget. The third is that execution-filtered distillation matches our method per unit of cost on held-out languages.

One outcome would strongly support the proposal. That is a clear held-out-language gain over all three automated arms, concentrated in Wrong Problem Mapping errors and in the rarest language classes, with non-programmer annotators reaching correction quality similar to programmers at a fraction of the cost.