Flag, Don't Guess: Ambiguity-Aware Translation of Non-English Science

A large body of science written in Russian, Chinese, Spanish and other languages reaches English readers only through machine translation. The original authors are often unavailable to explain what they meant, and English readers cannot check the source. When a source sentence has two grammatical readings, a translation system picks one and renders it as fluent, confident English. An example is a Russian sentence in which a condition could modify either the baseline or the intervention. If the system picks the wrong reading, nothing on the page signals the error. We propose to measure how often this silent commitment happens in scientific translation and in which direction it goes. We also propose a translation procedure that commits to a reading only when the rest of the document supports it. Otherwise it preserves or explicitly marks the ambiguity.

The motivation comes from two findings that have not been connected. Research on regional isolation shows that non-English works are cited for real methodological reasons but are nearly invisible to indexing. Cross-Lingual Citations in English Papers finds that cross-lingual citations have the same intent distribution as monolingual ones, yet only 11.2% of them resolve in the Microsoft Academic Graph, against 42.6% overall. Separately, Who Relies More on World Knowledge and Bias for Syntactic Ambiguity Resolution shows that frontier LLMs prefer "low attachment" across six languages. Low attachment means attaching a relative clause to the nearest noun. The models keep this preference even for Russian and Spanish, whose speakers more often prefer "high attachment" to the earlier noun. Taken together, these findings predict a specific, testable harm. LLM translators of Russian or Spanish science should resolve attachment ambiguities in an English-default direction. This would systematically misstate some regional claims to exactly the audience that cannot check them.

Core Claim

We make two claims. First, when translating non-English scientific text whose authors cannot be consulted, current systems make wrong commitments on genuinely ambiguous sentences that go unflagged. These errors are skewed toward English attachment defaults rather than source-language defaults. Second, a translator that works in three steps reduces these silent errors at equal cost to a frontier LLM given the full document and told to preserve ambiguity. The three steps are to write out the candidate readings explicitly, test each against the surrounding document (equations, tables, later sentences), and commit only when one reading clearly wins. The key ingredient is explicit enumeration. We expect a model that generates freely to resolve ambiguity before it notices the ambiguity exists. This bet can fail, and the experiments are designed to show whether it does.

The claim is deliberately narrow. We do not argue that ambiguity-preserving translation should be the default for all science. When the author is present, asking the author is better than any algorithm. The regime we study is translating existing regional literature for outside readers, which is the regime the isolation literature shows matters.

Relation to Prior Work

Symbolic ambiguity-preserving translation supplies the conceptual target but not a usable method. Ambiguity Preserving Machine Translation using Packed Representations and Syntactic and Semantic Transfer with F-Structures carry alternative attachments through transfer in a single packed grammatical structure. Semantic-based Transfer keeps scope and attachment underspecified in a semantic representation. Preserving Ambiguities in Generation via Automata Intersection defines a "safe" translation, meaning one valid under every source reading, and finds such strings by intersecting the target-language realizations of each reading. We reuse their idea that each reading should be represented explicitly and that a target string can be checked against all readings. We drop their hand-built grammars.

The same Knight et al. paper also constrains the design. It reports that safe strings are often disfluent. More importantly, it reports that readers of a technically ambiguous string are often pushed toward one reading anyway. This is why our procedure can also mark alternatives rather than only emit a safe string. An English rendering of a Russian relative clause may stay grammatically ambiguous while English readers still read it with low attachment. A preserved string is therefore not the same as a preserved ambiguity, and our reader study measures the difference.

On the neural side, Jam or Cream First? SCONES replaces the softmax output with independent per-token sigmoids. This lets several translations score highly at once, and it reports 1–9% relative BLEU gains on WMT19 and stable quality at beam size 1000. SCONES models uncertainty over target wording, but it never tests whether that uncertainty tracks distinct source readings, and it never touches scientific text. We treat it as a secondary baseline for detecting readings, not as the mechanism.

On the isolation side, PDFMathTranslate translates scientific PDFs while preserving layout. It detects equations and tables and sends text blocks to single-output engines, and it reports speed (1.47 s/page) but no translation-quality numbers. We reuse its segmentation to recover the equations and tables our procedure tests readings against. Translate your Own supplies the cost model. Domain experts post-editing commercial MT of NLP abstracts reach an HTER of 18.2, against 8.0 for professional translators. HTER, human-targeted translation edit rate, is the fraction of words a human changes. Our comparison adds a question this cost model leaves out: how many errors survive post-editing because nothing marked them.

Research Program

We first build an evaluation set of real ambiguous scientific sentences, then run the measurement study, then build and test the procedure.

To build the set, we use unarXive citation data (unarXive 2022) and the intent labels of Saier et al. to identify Russian-, Chinese- and Spanish-language works cited by English papers for their methods or results. Obtaining their full text is an engineering step outside unarXive itself, and we expect it to reduce the sample. We segment the documents with the PDFMathTranslate pipeline. Candidate ambiguous sentences are found by running a parser and an LLM detector and taking the union of their flags, which favors recall. Bilingual domain experts then label each candidate on three points: the readings it admits, whether the document resolves the ambiguity and in which direction, and whether a competent source-language reader would find it ambiguous at all. A document can resolve an ambiguity through an equation that defines the variable, a table header, or a later sentence. Where contactable authors exist, we ask a subset directly, which calibrates the "document-resolved" labels. We target a few hundred confirmed ambiguities per language, spread across relative-clause attachment, prepositional-phrase attachment, quantifier and negation scope, and term polysemy. We include matched unambiguous sentences to measure false alarms.

To make the attachment finding comparable to controlled psycholinguistics, we also translate the ambiguous condition of MultiWho into English. We record which attachment each system's output commits to, and whether its output leaves the ambiguity open.

The procedure has three stages. In the first, it lists readings. A model is asked to produce, for a source sentence, a set of unambiguous English paraphrases, one per grammatical reading, or a single paraphrase if there is only one reading. In the second, it scores each reading against evidence gathered from the document: the equations near the sentence, table and figure captions, and a window of preceding and following sentences. An entailment-and-contradiction model does the scoring. In the third, it decides:

$$m = s_{(1)} - s_{(2)},$$

where $s_{(1)}$ and $s_{(2)}$ are the evidence scores of the best and second-best readings. If the margin $m$ exceeds a threshold $\tau$, the translator commits to the best reading. Otherwise it tries to generate a safe English string. It does so by asking for one rendering and checking, with the same entailment model, that the rendering is consistent with every listed reading. If no safe string passes, it emits the best-supported reading with a short inline marker giving the alternative. We set $\tau$ on a held-out split and report a curve of results as $\tau$ varies, not a single point.

Baselines

The naive reference point is single-output commercial translation run through the PDFMathTranslate pipeline with DeepL. This is the tool a reader would actually use, and Translate your Own found DeepL the lowest-edit engine, at HTER 7.1 for translators and 13.8 for experts. We run it on the identical document set. It beats our procedure if its silent-wrong-commitment rate is no higher.

The strongest competitor is a frontier LLM given the entire source document and instructed to translate faithfully and to keep or flag genuine ambiguities. We run the same model family that our procedure uses internally. We give it at least the same total token budget as our three-stage procedure, and we let both sides use the same held-out split for prompt tuning. This baseline has the same information and the same explicit instruction as our procedure, so it isolates whether structured enumeration matters. Our procedure beats it only if its silent-wrong-commitment rate is lower at equal or better fluency and equal or lower false-flag rate. We will report a difference as meaningful only if a bootstrap confidence interval over sentences excludes zero. The file contains no reported result for this configuration on any ambiguity-preservation metric. For the related forced-choice setting, MultiWho reports 95–100% LLM accuracy when world knowledge points to one reading, together with a rigid low-attachment preference when it does not. We have found no Lacuna-reported result for ambiguity preservation in scientific translation.

Two further baselines test cheaper ways to detect ambiguity. The first is sampling disagreement: sample several translations at nonzero temperature and flag a sentence when the translations commit to different readings. The second is a SCONES-style multi-label translation model, tested on whether its top alternatives separate the listed readings. The multi-label model would need retraining on in-domain data, so we run it where training data allow and state plainly if we cannot.

Two ablations isolate the ingredients of our procedure. One removes enumeration, so the model translates with the same document evidence but no explicit reading list. The other removes grounding, so the procedure lists readings but decides without document evidence, using the model's own preference.

Why not just…?

Why not just give a frontier LLM the whole document and ask it to preserve ambiguity? This is the real threat. If long context plus instruction already drives silent wrong commitments near zero, our procedure adds nothing, and the contribution reduces to the measurement study. The MultiWho result suggests models resolve before they notice, since their attachment preference does not change with the language. But that study used forced choice on isolated sentences, not translation with context. The no-enumeration ablation, which uses identical evidence and budget, settles the question directly. We will report a null result as a null result.

Why not just sample several translations and flag disagreement? Sampling is cheap and needs no new components. However, it detects ambiguity only if the model's samples actually spread across readings. A model with a rigid low-attachment preference will produce samples that all agree on the wrong reading, and we predict low recall on attachment ambiguities specifically. The sampling baseline, broken down by ambiguity type, tests this. If sampling matches enumeration on recall, we will adopt it as the detection stage.

Why not just let the reader or a post-editor fix errors afterwards? Post-editing works when something looks wrong. A fluent wrong commitment looks right, and the English reader of a Russian paper has no source to compare against. The post-edited HTER figures in Translate your Own measure edits made, not errors missed. Our expert post-editing condition measures how many labeled wrong commitments survive editing, with and without markers.

Why not just produce a safe, ambiguity-preserving English string everywhere? Preserving Ambiguities in Generation via Automata Intersection reports two problems with safe strings: they are often disfluent, and readers are pushed to one reading anyway. Our reader study includes a safe-string-only condition to test whether preserved strings actually keep the ambiguity open for English scientists.

Proposed Experiments

The measurement study comes first. For each system, language and ambiguity type, we report three rates: silent wrong commitment (the output commits, unmarked, against the reading the document supports), unwarranted commitment (the output commits, unmarked, where the document does not resolve the ambiguity), and false flags on the matched unambiguous controls. For relative-clause cases the document does not resolve, we compare the fraction of low-attachment commitments against native-reader preferences for the source language. The MultiWho human data supply those preferences where the languages overlap, and a small native-speaker judgment study supplies them elsewhere. The decisive prediction is that commitments from Russian and Spanish lean toward low attachment more than native readers of those languages do. If they do not, the isolation-specific harm we hypothesize is absent, and we will say so.

The method comparison is second. We plot silent wrong commitment against false-flag rate over $\tau$ for our procedure, and against the corresponding knob for each baseline. Fluency is rated by English-speaking scientists who do not see the source. Our procedure must dominate the full-context LLM baseline on this curve at comparable fluency to count as a result. The two ablations indicate whether enumeration, grounding, or both carry any gain.

The reader study is the final and most decisive check. English-reading scientists in the relevant field read translated passages and answer a comprehension question that hinges on the ambiguous claim. They also give a confidence rating. We compare four conditions: plain DeepL output, the full-context LLM, safe strings only, and our full procedure with markers. The outcome that matters is the rate of confident wrong answers, since that is the error that would propagate a misreading of regional work into English citation. A secondary outcome is time and perceived burden, because markers that readers find tiresome would not be adopted.