Reading to Repair: Using a Model's Comprehension to Fix Its Own Low-Resource Translations

Large language models read low-resource languages much better than they write them. The same model that correctly classifies sentiment in Indonesian regional languages produces hallucinated or wrong-language output when asked to translate into them (Bang et al.). It answers questions about Indic text well in English and then degrades when it has to answer in the native script (IndicGenBench). It solves linguistic puzzles from an unknown language into English far better than in the reverse direction (Zhu et al.). The usual response treats this gap as a deficit to train away. We propose using it at inference time. The model's stronger hand, reading, can check the work of its weaker hand, writing.

The proposal is a small team of agents built from one model. A writer drafts the English-to-target translation. A reader sees only the target-language draft, never the English source, and produces a literal word-by-word English gloss. A comparator works entirely in English. It sets the gloss against the source, names concrete mismatches such as a missing plural, a wrong tense, or a dropped negation, and maps each mismatch back to a span of the draft. The writer then repairs only those spans. The claim is that this targeted repair, which uses the model's stronger direction as the error detector, beats whole-sentence resampling and self-critique at equal compute. The advantage should grow with how many tokens the target language needs per sentence.

Core Claim

Translating from English into a low-resource language fails because of error accumulation during generation. The model must emit many tokens from an uncertain distribution, and target words are split into many subword pieces (IndicGenBench). A simple model makes the consequence concrete. Suppose each emitted token is wrong with probability $\epsilon$ and a translation needs $L$ tokens. Then an independent sample is error-free with probability roughly $(1-\epsilon)^L$. Best-of-$N$ sampling therefore needs $N$ on the order of $(1-\epsilon)^{-L}$ to contain a clean candidate, which grows exponentially in $L$. Localized repair needs work proportional to the expected number of errors, about $\epsilon L$, which grows linearly. This model is crude because errors are not independent. Still, it yields a falsifiable prediction: the gain of targeted repair over sampling-and-reranking should widen with tokenizer fertility (tokens per word) and sentence length. In the reverse direction, into English, the gain should be small, because there the writing side is not the bottleneck.

Two design choices separate this from generic multi-agent critique. First, the critic's evidence comes from the model's strong direction, comprehension of the target language rendered into English, rather than from the model judging target-language text directly, which is where its judgment is weakest. Second, the reader is blind to the source. A reader that sees the English already "knows" what the draft should mean and will read that meaning into broken text. This is the confirmation bias that motivates separated roles in Self-collaboration Code Generation and MACM.

How This Relates to Prior Work

Several papers supply the motivation by establishing the comprehension–production gap at inference time: Bang et al., IndicGenBench, IndoNLG (generation is more sensitive to pretraining volume than comprehension), and Zhu et al.. None of them turns the gap into a mechanism.

Two papers supply method. MACM separates a proposer from a judge that admits only verified conditions, which raised GPT-4 Turbo on Level-5 MATH from 54.68% to 76.73%. We borrow its rule that nothing enters the output without passing a check. Self-collaboration shows that an isolated tester role catches errors the coder misses. Our reader plays the tester. Its "test" is a back-reading into English.

The closest work is TransAgents, a six-role "translation company" for long literary text between high-resource languages. It applies the same pipeline in both directions, never measures low-resource targets, and underperformed simpler baselines on short texts. That last result constrains our design, because FLORES-style benchmarks are short sentences. We therefore use three roles, no preparation phase, and a hard per-sentence budget.

Zheng et al. report that English-to-target fine-tuning is synergy-dominated and target-to-English fine-tuning is conflict-dominated. They address this with direction-specific adapters. That finding seems to cut against ours, since it suggests English-to-target is the easier direction to train. We read the two results as describing different things: shared structure helps the parameters during training, while emitting low-resource tokens at run time still fails. Our method needs no parallel training data, so it is complementary. We test it on top of a fine-tuned translator as well as on a general model.

OmnilingualGAIA2 adds a caution. Agents lose subtle constraints such as definiteness and plurality when parsing non-English input. Those are exactly the cues our reader must preserve. For this reason the reader is asked for a literal gloss, not a fluent paraphrase.

Research Program

The system uses one base model in three isolated contexts. The writer receives the English source and produces a draft. The reader receives only the draft. It returns an English gloss aligned to draft spans, and it flags spans it cannot parse. The comparator receives the source and the gloss. It returns a short list of typed mismatches (meaning missing, added, or altered, by grammatical category), each tied to a draft span. The writer receives the draft, the flagged spans, and the typed complaints, and rewrites only those spans. An acceptance rule prevents the loop from spreading errors: a repair is kept only if a fresh reader pass on the new draft yields fewer comparator mismatches. Otherwise the previous draft stands. The loop stops when no mismatches remain or the token budget runs out.

Before any translation result, a calibration study asks whether the reader can detect errors at all. This is the central risk. Comprehension being robust to noise may make the reader too forgiving, so that it "understands" ungrammatical text. We will corrupt FLORES-200 reference translations with controlled edits (agreement flips, dropped negation, number changes, word-order swaps, wrong-language substitutions) and measure the reader-plus-comparator's span-level detection precision and recall. We compare against a direct judge in the target language and a reference-free neural quality estimator. If blind reading does not beat direct target-language judging on detection, the project's premise fails, and we would report that.

The main study runs English-to-target translation across languages chosen to span tokenizer fertility and resource level. It covers several base models, including open 7–13B models. Self-collaboration notes that models below about 13B struggle with role protocols. Our roles are narrow and structured, and whether this holds at small scale is itself a result. The analysis regresses each method's gain on fertility and length to test the scaling prediction. It also runs the same pipeline in the target-to-English direction as a control.

Baselines

The naive reference point is one-pass translation by the same base model with a standard few-shot prompt. This shows how much any extra computation buys.

The most dangerous competitor is round-trip reranking at matched compute. The model samples $N$ target-language candidates and back-translates each into English. The candidate whose back-translation best matches the source, by an English-side similarity score, is kept. This baseline already exploits the same asymmetry. The only difference from our method is that it selects whole candidates instead of repairing spans. We set $N$ so that total generated tokens equal the median spend of our loop. We tune its prompt and scoring with the same development budget on the same FLORES-200 dev split. Beating it means a significant ChrF++ gain on the test split, averaged over high-fertility languages, together with a gain in human error counts. Matching it overall but beating it only at high fertility would still support the scaling claim.

The strongest generic agentic baseline is single-context self-refinement. The same model critiques and revises its own translation, with and without seeing the source, at matched tokens. We also include a TransAgents-style multi-role pipeline (TransAgents) cut down to sentence level. Reported figures for it exist only for literary Chinese (GEMBA-DA 87.7 versus 85.8 for GPT-4-Turbo, d-BLEU 25.0), which are not comparable to our setting. We will rerun it rather than borrow numbers.

As dedicated translation systems we run NLLB-200 (No Language Left Behind) and X-ALMA (X-ALMA), and we apply our loop on top of each. X-ALMA reportedly outperforms Aya-101, Aya-23 and NLLB-3.3B in every FLORES-200 direction by COMET-22, but the file gives no per-direction values. No reported result was found on Lacuna for X-ALMA's or NLLB's English-to-low-resource scores on FLORES-200. We will measure both ourselves. For context, FLORES-101 reports that very low-resource languages often scored below 3 spBLEU (FLORES-101).

Three ablations isolate the key ingredients. The first lets the reader see the source; this tests blindness. The second replaces the gloss-in-English critic with a critic that judges the target text directly in the target language; this tests whether the strong direction matters. The third keeps detection but discards the localization, regenerating the whole sentence with the complaints attached; this tests whether targeted repair matters.

Why not just…?

Why not just do round-trip reranking, which also uses the strong direction? This is the real threat, and it may match us on short, low-fertility sentences. Our error-accumulation argument predicts it falls behind specifically as tokens per sentence grow, because clean whole candidates become exponentially rare. The fertility-stratified comparison at matched tokens settles the question. If no widening gap appears, we would report that reranking suffices and that the repair machinery is unnecessary.

Why not just let the model self-correct in one context? Prior agent work found single-context self-review blind to its own errors: Self-collaboration reports a 29.9% Pass@1 gain on HumanEval from an isolated tester, and LEGO reports that removing its critic hurt. Those findings come from code and causal explanation, not translation. Our blind-reader ablation and the self-refinement baseline test whether the effect transfers.

Why not just use a dedicated translation model like NLLB or X-ALMA? These are stronger writers, and we do not expect a general model with agents to beat them outright on every language. The relevant question is whether the reading loop improves their output too, since detection only needs a capable reader. Running the loop on top of NLLB and X-ALMA answers it. The method also applies where no parallel data exists, as in the unseen-language puzzles of Zhu et al..

Why not just pivot through English or use step-by-step curriculum prompting? Pivoting does not help when English is already the source. Direct translation also beat pivoting in 80% of Indic directions on FLORES-101 (FLORES-101). Curriculum prompting raised GPT-4o puzzle exact match from 6.8% to 22.6% (Zhu et al.), but its stages were hand-built by linguists. We include it as a baseline on the puzzle set, and we combine it with our loop to see whether the gains add.

Proposed Experiments and Decisive Checks

Translation is evaluated on FLORES-200 English-to-target for roughly a dozen languages spanning low to high fertility, and on IndicGenBench's FLORES-IN translation task. We also run Indonesian-to-Sundanese and Indonesian-to-Javanese from IndoNLG's translation data, which tests the method with a non-English but stronger source. Finally, we use the English-to-unknown direction of Zhu et al.'s 168 puzzles. That paper reports 82.0 ChrF for Claude 3.5 Sonnet only in the easy unknown-to-English direction. No reported result was found on Lacuna for the English-to-unknown direction, which we will measure.

We report ChrF++ and spBLEU as primary metrics. COMET is secondary, because the file notes that neural quality estimators are less reliable when the target is low-resource. On a stratified sample we add native-speaker annotation of span-level errors, which also audits whether the comparator's flags were real.

Four results decide the project.

The calibration study comes first. Blind reading must detect injected errors better than direct target-language judging, or the premise fails.

The fertility interaction comes second. The gain over matched-compute round-trip reranking must grow with tokens per word and with sentence length.

The direction control comes third. Running the same loop into English should yield a markedly smaller gain, which confirms that the method exploits the asymmetry rather than generic extra computation.

The ablation pattern comes fourth. Removing blindness, removing the English-side critic, or removing localization should each cost accuracy. If only the critic matters, the method collapses into a reranker.

As a guard on cascading errors, we track the fraction of accepted repairs that annotators judge harmful. A rising rate across loop iterations would indicate the acceptance rule is too loose.