Sense Unit Tests: Execution-Style Repair for Polysemous Words in Translation

When a code generator writes a buggy function, the best current systems do not just ask the model to "try again." They run the code, capture the specific failing assertion, and hand that error back to the model. Machine translation has nothing comparable for one of its most persistent failures: picking the wrong meaning of a word that has several. An example is translating the Italian-bound English word bank with the river sense when the sentence is about finance, or the reverse. Current LLM translation refiners either polish the whole sentence without any checkable target, or score drafts with a learned quality estimator that returns one noisy number. We propose to build the missing oracle. For each ambiguous source word, a "sense unit test" looks up its candidate meanings in a multilingual sense inventory. The model then picks the intended meaning in a question-answering step, where LLMs are known to be more reliable. Finally, the test deterministically checks whether the word the translation actually used belongs to that meaning's set of valid target-language words. A failed test yields a concrete error message, in the style of a compiler error, which drives a targeted repair. The loop reverts any edit that breaks a previously passing test.

The opportunity is sharp because the failure is well measured and has a specific shape. On DiBiMT, the best commercial system reaches only 60.7% mean accuracy, and nearly 90% of errors choose a more frequent sense than the context requires (DiBiMT). Meanwhile, LLMs often do know the right sense when asked directly, yet fail to use that knowledge while translating (Aligning Translation-Specific Understanding to General Understanding in Large Language Models). That gap between knowing and producing is exactly what execution-style verification addresses in code. A model that "knows" what the function should do still benefits from a test that tells it where its output diverged.

Core Claim

Our claim is narrow and falsifiable. For rare senses of polysemous words, a translation refiner driven by a discrete, word-localized, retrieval-backed check will beat two alternatives. The first is a refiner driven by scalar quality estimation. The second is a one-shot translator that is simply told the correct sense in advance. The mechanism has three parts: retrieval supplies the sense inventory and target lexicalizations that the model's weights may not surface, the question-answering-mode choice supplies the intended sense, and the post-hoc test catches cases where the generator ignores that choice. If the one-shot "told in advance" baseline matches the loop, the contribution reduces to the sense-selection step and we will report it that way. The experiments are designed so that this outcome is distinguishable.

How This Differs From the Closest Work

Three papers sit closest. DUAT flags difficult words, has the LLM write target-language explanations of them from its own memory, and keeps explanations only if a sentence-level quality estimator predicts improvement (Aligning Translation-Specific Understanding to General Understanding in Large Language Models). It motivates our work, because it establishes the understanding-versus-translation gap. Our design differs from it on two axes. DUAT has no external retrieval, so a sense the model never learned cannot be recovered. Its feedback is also a single scalar per sentence, not a check on the specific word. APIRAT, the method source, runs a translate-test-refine loop for code: failing unit tests trigger retrieval of API usage examples and mappings, which are fed into the repair prompt (APIRAT). We transplant its "failure triggers retrieval" structure to natural language, where no test suite exists, so we must construct the test. Iterative Translation Refinement shows that zero-shot LLM polishing improves human preference, but it drifts in meaning without source anchoring and never targets individual words (Iterative Translation Refinement with Large Language Models). We keep source anchoring and replace open-ended polishing with edits confined to failing words.

The code refinement literature also constrains the design. Compiler-feedback loops plateau after a few rounds (Yan et al.), and unconstrained iterative prompting can amplify defects (Srivastava et al.). We therefore cap iterations and borrow the idea of reverting to the last stable checkpoint from rollback-based hardware code repair (Fang et al.). The code-reasoning survey names our target explicitly. It notes that execution feedback breaks down where "executability" is ill-defined, as in natural language semantics (Code to Think, Think to Code). We do not claim to make translation executable in general. We define a small subset of it, lexical choice for inventory-covered words, that can be checked mechanically.

Research Program

The system has four components. The first is sense retrieval. For each content word in the source, we query BabelNet, a multilingual semantic network that links WordNet senses to words in many languages (Navigli and Ponzetto). We retrieve every candidate sense, its gloss, and its target-language lexicalizations. We also retrieve a few parallel example sentences per sense from OPUS, the same resource MuCoW was built from (MuCoW). Only words with at least three senses and divergent target lexicalizations get tests, which keeps cost proportional to ambiguity.

The second component is sense selection in question-answering mode. The LLM sees the source sentence and the numbered glosses and picks one. This step exploits DUAT's finding that understanding is better outside translation mode. It is also consistent with probing evidence that contextual encoder states resolve word senses with over 91% accuracy (Encoders Help You Disambiguate Word Senses in Neural Machine Translation). The frequency trap is the main risk here, since the model may still favor the common gloss. For that reason, glosses are presented in shuffled order, with no frequency information.

The third component is the test itself. The system translates, aligns the source word to target tokens with an off-the-shelf multilingual word aligner, lemmatizes the aligned span, and checks whether it belongs to the lexicalization set of the selected sense. The test returns pass, fail (the word belongs to another sense's set), or unknown (the word is in no set, possibly a valid synonym that is missing from the inventory). On fail, the error message names the chosen word, the sense it actually expresses, the intended gloss, the permitted target words, and one retrieved target-language example.

The fourth component is repair with rollback. The model edits the draft given the failing messages, and all tests are rerun. If any previously passing test now fails, or a quality estimator's score drops beyond a threshold, the edit is reverted. The loop stops when all tests pass or after three rounds.

We build this on two open models, one around 7B parameters and one larger, so that the effect of scale is visible. We release the test harness separately so it can serve as a diagnostic.

Baselines

The naive reference point is direct one-shot translation by the same LLM with the same few-shot examples. This establishes the frequency-trap error rate we start from. As an external anchor, we quote the strongest reported DiBiMT number: DeepL at 60.7% mean accuracy, with Google Translate, MBart50 and OPUS between 20% and 33% (DiBiMT). We will not rerun commercial systems. That number fixes the bar for "useful": a 7B open model with sense tests should approach or pass 60.7% on the same items.

The strongest existing method is DUAT, which targets exactly this failure with an iterative, quality-controlled loop. We reimplement both variants, the intrinsic one (DUAT-I) and the external quality-estimator one (DUAT-E), on the same models, data, and quality estimator. Every method gets the same cap on LLM calls per sentence, so the comparison is not won by extra compute. DUAT's reported result is up to +3.85 COMET over in-context-learning baselines on Challenge-WMT English-to-Icelandic, along with a 25% to 51% reduction in literal translation errors under human evaluation (Aligning Translation-Specific Understanding to General Understanding in Large Language Models). We found no reported DUAT result on Lacuna for DiBiMT or MuCoW, so those comparisons will be new. Beating DUAT means higher DiBiMT accuracy and higher MuCoW F1 in the under-20%-frequency bin at equal call budget, with Challenge-WMT COMET no worse than DUAT's.

The third baseline is unconstrained iterative polishing with source anchoring, following Iterative Translation Refinement (Iterative Translation Refinement with Large Language Models), run for the same number of rounds.

The key ablations isolate each ingredient. "Hint-only" gives the translator the selected sense's gloss and permitted target words before translation, with no test and no loop. It asks whether verification adds anything beyond retrieval-plus-selection. "Test-as-reranker" samples as many candidates as the loop's call budget allows and returns the one passing the most tests, which separates the value of the check from the value of guided repair. "Oracle sense" replaces the model's choice with the gold sense and bounds how much error comes from selection versus repair. "No retrieval" has the model write its own sense list, DUAT-style, and feeds it into the same test. This isolates what the external inventory contributes, which matters most for rare senses. "No rollback" measures regressions.

Why not just…?

Why not just tell the model the right sense up front and translate once? This is the most serious threat, and the hint-only ablation is built to settle it. Our reason to expect a gap is the evidence that models fail to act on knowledge they have. DUAT documents literal translations of words the model can define. DiBiMT reports that in 93% of error cases the model assigns higher probability to the incorrect translation, a bias a prompt hint may not override. A post-hoc check catches exactly the cases where the hint is ignored. If hint-only matches the full loop on the low-frequency bins, we will say so. The contribution would then reduce to retrieval-backed sense selection, which is still new relative to DUAT.

Why not just use DUAT with a stronger quality estimator? A sentence-level score is a poor detector of single-word sense errors. A fluent sentence with one wrong noun can score well, and DUAT's own analysis shows filtering by such scores is needed precisely because unfiltered interpretations add noise. We run DUAT-E with the same quality estimator we use for rollback. If our loop wins, the gain is attributable to word-level checks and external senses rather than to a better scorer.

Why not just sample many translations and pick the best by quality estimation? Best-of-N with the same call budget is included, with two selectors: quality estimation, and our tests. If test-based reranking equals the repair loop, then the check, not the iterative repair, is the active ingredient, and we would report the cheaper method. The code refinement literature does suggest that exploring and then selecting candidates can compete with iterative repair (SwiftTrans).

Why not just use a bigger model? DeepL's 60.7% ceiling on DiBiMT shows that scale and engineering alone leave a large rare-sense gap. MuCoW shows F1 collapsing to 40% to 70% for senses under 20% of training frequency, and more web-crawled data made it worse, not better (MuCoW). Running both model sizes tells us whether the loop's gain shrinks with scale. If it vanishes at the larger size, that is a meaningful negative result.

Proposed Experiments

The primary benchmark is DiBiMT: 600 items, 305 lemmas, English into Chinese, German, Italian, Russian and Spanish. We report accuracy, the rate of cases where no valid translation of the target word is produced, and the benchmark's frequency- and polysemy-sensitivity indices, split by nouns and verbs, since verbs showed 18 points more misses. MuCoW supplies the rare-sense analysis: F1 per training-frequency bin, where our method should show its largest gains in the lowest bin. Challenge-WMT gives sentence-level COMET on hard sentences for comparison with DUAT. WMT22 News COMET checks that confining edits to failing words does not degrade ordinary text, and it tracks the fluency cost that Iterative Translation Refinement observed. We also report LLM calls, tokens, and latency per sentence, and the fraction of sentences that trigger any test.

Four checks decide whether the claim holds. First, the full loop must beat hint-only on DiBiMT and on MuCoW's lowest frequency bin. Otherwise verification adds nothing. Second, the gap between the oracle-sense and predicted-sense runs tells us whether sense selection is the bottleneck. If it is large, the next step is better selection, not better repair. Third, the checker itself must be audited for two problems. One is circularity: DiBiMT's good and bad candidate lists are derived from lexical resources that overlap with BabelNet, so we measure that overlap and additionally evaluate a human-judged sample. The other is false alarms: bilingual annotators will label a sample of "fail" and "unknown" verdicts to estimate how often valid synonyms are wrongly flagged. Fourth, the no-rollback ablation should show regressions across rounds. If it does not, the rollback machinery borrowed from code repair is unnecessary for this task. We will report all four outcomes, including the ones that shrink the claim.