Measuring What the Translation Lost: Source-Reconstruction Signals for Translate-Then-Parse Pipelines

A common way to serve users in many languages is to translate their input into English and hand it to a strong English system, such as a parser that turns questions into database queries (SQL) or a tagger that finds names in text. This is called translate-test. When the translation drops or distorts something, for example losing the word "average" from a database question, the English system never sees it and produces a confidently wrong answer. The error cascades: a mistake in an early stage is silently passed to later stages as if it were true. Current fixes feed the downstream model the original sentence alongside the translation on every input. These fixes work, but they pay the cost of joint processing everywhere. They also leave open whether the problem is structural or just a symptom of poorly tuned components.

This note proposes to measure, for each word of the source sentence, how much of it the translation failed to carry. The instrument is a bilingual masked language model: a model that reads the source sentence and its translation side by side and tries to fill in hidden source words. If a source word is easy to recover from the translation, the translation kept it. If it is hard to recover, the translation lost it. The central bet is that cascading pipeline errors are concentrated on exactly these hard-to-recover words. A pipeline that patches only those source spans back into the downstream input should then recover most of the benefit of full joint processing at close to translate-test cost. The same measurement lets us settle an open dispute about whether joint modeling is really needed once each component has been fine-tuned.

Core Claim

We make three testable claims. First, per-token source-reconstruction loss from a bilingual model predicts which instances a translate-test pipeline will get wrong, and it does so better than the translation model's own confidence. Second, the accuracy gain that joint bilingual methods achieve over translate-test is explained mostly by instances with high reconstruction loss. Third, this gain survives in-domain fine-tuning of both components, which a recent speech-translation study found erases the advantage of joint training.

The method that follows from these claims, which we call targeted bilingual patching, does not concatenate the full source everywhere. It appends only the high-loss source spans, each marked with its position in the source, to the translated input.

This is our reading of the bilingual language modeling tradition. That tradition argued that predicting target words from a joint source–target window reduces ambiguity, as in the bilingual n-gram tokens of Niehues et al. and the neural joint model of Devlin et al.. We flip the direction. Instead of using bilingual context to pick target words, we use it to score how well the source survives given the target. That score then serves as a per-token diagnostic of information loss at a pipeline boundary.

Relation to Prior Work

Several papers motivate the gap. Shi et al.'s REX encodes the native-language question, its English translation, and the schema together, then mixes their hidden states. It reports 59.7% exact match on CSpider and only a 1.1% drop under a weaker translator, against up to 4.1% for simpler baselines. TransFusion translates low-resource text to English, annotates the English, and then has a 7B model fuse the annotations with the original text. This raises MasakhaNER 2.0 F1 from 47.9 to 62.4.

Both methods apply joint conditioning uniformly and require custom training or multiple model calls. Neither says which inputs actually needed the source. We keep their premise, that the source contains information the translation drops, and turn it into a per-token measurement and a selective intervention.

IBUT generates bilingual natural-language explanations and reconciles them. It resolves about 89% of "understanding distortion" errors but costs about 5x the tokens, and it addresses translation itself rather than downstream tasks. Our signal is a single forward pass of a small model, not a multi-turn loop.

Other papers supply parts of the method. The bilingual masked objective comes from work that masks tokens across concatenated sentence–translation pairs (Li et al., Unicoder). Fan et al.'s Bilingual Expert shows that cross-lingual "surprise" features from a bidirectional model detect translation errors well. It reports a Pearson correlation of 0.7159 on WMT17 En-De sentence-level quality estimation. However, it scores target tokens for offline evaluation. We score source tokens for omission and feed the signal back into a live pipeline.

Kumar et al. coupled speech recognition to translation using phrase-translation entropy and gained only 0.2 BLEU. Their entropy measures the uncertainty of the downstream model, which by construction cannot detect a confident omission. That is precisely the failure we target.

Two papers constrain the design. Tran et al. show that on MuST-C, fine-tuning speech recognition and translation in-domain lifts the plain cascade from 29.0 to 30.5 BLEU and removes all gains from joint training and soft integration. Any claim that bilingual context is structurally necessary must therefore be tested against fine-tuned components, and we build that test in. Wu et al. show that where errors fall in an output depends on the language's branching structure, not only on errors carried forward during generation. We therefore expect where information is lost to differ between Japanese and German sources, and we analyze loss by language rather than pooling.

Research Program

The instrument is a multilingual encoder, for example XLM-R-large or an mT5 encoder, trained with translation-pair masking. Given a source sentence and its translation concatenated, it masks spans only on the source side, with the translation always visible. Training uses human parallel data.

At test time we mask each source word in turn, with translation and the rest of the source visible. The loss for word $i$ is

$$\ell_i = -\log p_\theta(s_i \mid s_{\setminus i}, t) + \log p_\theta(s_i \mid s_{\setminus i}),$$

where $s$ is the source sentence, $t$ the machine translation, $s_{\setminus i}$ the source with word $i$ masked, and $p_\theta$ the bilingual model. Both terms use the same model; the second term simply hides the translation as well.

Subtracting the monolingual term gives a measure of how much the translation helps recover the word. This separates two cases. A word that is simply predictable from its own sentence, such as a function word, needs no help from the translation. A content word that the translation should have explained, but did not, is the case we want to flag. A word is flagged when $\ell_i$ exceeds a threshold tuned on a development set. The instance-level score is the maximum $\ell_i$ over content words.

Patching works as follows. For flagged words we append the source span, which is the word plus one neighbor on each side, with a short tag, to the translated question given to the downstream system. For SQL we use an off-the-shelf English parser. For NER we use TransFusion's annotation model on the English side and project spans back only for patched regions. If an instance has no flagged spans, it runs as plain translate-test.

We will also train a simple oracle-labeled variant. On MultiSpider, each question exists in seven languages with the same SQL, so for every non-English question we can run the English parser both on the machine translation and on the gold English question. Instances that are right on gold English but wrong on the machine translation are, by definition, cascade failures. This gives exact labels for how well $\ell$ detects cascade failures, with no human annotation.

Baselines

The naive reference point is plain translate-test. We translate with NLLB-200 and, separately, with a weaker Marian model, then run the same English parser or tagger with no source access. Alongside it we report the oracle, the same downstream system on gold English, which bounds how much cascade loss exists at all. For CSpider, FastRAT reports zero-shot ChatGPT at 32.6 exact match as a weak monolithic reference.

The strongest existing methods are full joint bilingual conditioning. On CSpider this means REX, which reports 59.7% exact match, and FastRAT, which reports 61.3 exact match. On MasakhaNER 2.0 it means TransFusion (GoLLIE-TF), which reports 62.4 average F1, against 54.2 for GPT-4. On MultiSpider we found no reported absolute accuracy on Lacuna for any joint method. The paper reports only the 6.1% English-to-other drop and a 1.8% gain from its schema-augmentation method SAVE. We will therefore build a matched full-concatenation baseline there: the same downstream model receives the whole source alongside the translation.

For fairness, every system shares the same downstream backbone, the same translator, and the same training data. Every system is also evaluated in two regimes: off-the-shelf components, and components fine-tuned in-domain, following Tran et al.'s protocol. Hyperparameter budgets are matched, and we count tokens and wall-clock time per query.

Patching "beats" full joint conditioning if it comes within one point of its accuracy while touching fewer than a third of instances. Patching loses if it trails by more than two points.

The key ablation replaces the bilingual reconstruction gate with four simpler gates while keeping patching identical. The four are: the translation model's own sequence log-probability; a Bilingual-Expert-style quality estimation score on the target side; round-trip back-translation similarity; and a random gate at the same patch rate. A second ablation drops the monolingual correction term in $\ell_i$.

Why not just…?

Why not just fine-tune both components in-domain, as Tran et al. recommend? This is the most serious threat, since it erased joint-training gains in speech translation. Our view is that fine-tuning reduces noise but cannot restore content a translator confidently omits. Shi et al.'s dropped "average" is that kind of error, and TransFusion's alignment collapse under reordering is similar. The experiment settles it directly. We measure the oracle-minus-translate-test gap and the patching gain after fine-tuning. If both shrink to within noise, we will report that cascade loss in text-to-SQL and NER is also largely a baseline artifact, which would itself be a useful negative result.

Why not just always give the downstream model the full source, as REX and TransFusion do? If full concatenation is equally accurate and cheap enough, patching's only advantage is cost. For a fine-tuned encoder like REX, the cost difference may be small. We expect accuracy parity at best, so the claim we stake is the cost–accuracy frontier for LLM-based downstream systems, where every query pays for extra context and extra calls. We also make a diagnostic claim: full concatenation cannot tell you which inputs needed the source. If patching loses more than two points to full concatenation, the selective design is refuted.

Why not just gate on the translator's confidence, a quality-estimation model, or round-trip translation? Confident omissions are low-entropy under the translator, so its log-probability should miss them. Target-side quality estimation of the kind Fan et al. built scores what is present in the translation, not what is absent from it. Round-trip back-translation is the most dangerous simple alternative, because it does see omissions. However, it costs a second translation and is confounded by the back-translator's own paraphrasing. The gate ablation compares all of these at matched patch rates on the oracle cascade-failure labels. If round-trip similarity matches reconstruction loss, we will adopt it and report that the bilingual model is unnecessary.

Why not just sample several translations and vote downstream? Finkel et al. showed that sampling beats k-best lists in annotation pipelines, but the gains were modest: +0.7 F-measure on in-domain role labeling and +1.2 out of domain. Sampling also multiplies downstream cost. More importantly, if the translator systematically drops a word, every sample drops it. We include five-sample voting as a baseline to test this.

Proposed Experiments

The primary testbed is MultiSpider. It is the right benchmark because it has fully translated schemas, seven typologically varied languages including left-branching Japanese, and parallel questions that give exact cascade-failure labels. CSpider and VSpider give comparison against published REX and FastRAT numbers. MasakhaNER 2.0 tests the regime where parallel data for training the reconstruction model is scarce. This is a real risk: the gate may be weakest exactly where omissions are most common. We will report gate quality as a function of the bitext available per language.

The metrics are exact match and execution accuracy for SQL, and entity F1 for NER. For the gate, the metric is area under the precision-recall curve against oracle cascade-failure labels. For cost, we report tokens and latency per query.

The first decisive check tests whether reconstruction loss predicts cascade failure. Its precision-recall area on MultiSpider oracle labels should clearly exceed that of translator log-probability and quality-estimation gates in every language.

The second decisive check is attribution. Split test instances by gate score. The gain of full joint conditioning over translate-test should concentrate in the top-scoring fraction. If gains are spread evenly, the claim that cascade loss is localized is false.

The third decisive check is the fine-tuning test. The first two checks must hold after in-domain fine-tuning, which directly answers the question Tran et al. left open for structured tasks.

A secondary analysis by language asks whether flagged words fall at different sentence positions in Japanese versus German sources, in line with Wu et al.'s branching effect. We also inject controlled translation corruptions of three kinds, deleting quantifiers or aggregation words, swapping entity names, and reordering clauses. These show which error types the gate catches and which it misses.