Linked Questions: Multi-Hop Probes for Document-Level Translation Evaluation

When a machine translation system translates a long document, it can get every sentence roughly right and still break the document. A pronoun can point to the wrong character. A technical term can drift between synonyms. A causal "therefore" can vanish, so two facts no longer connect. Segment-level metrics score each sentence against its own reference and never see these failures. The recent idea of grading a translation by whether a reader can answer comprehension questions from it only checks local facts, so it misses them too. We propose to evaluate document translations with multi-hop questions. These are questions built so that the answer exists only if the reader connects a fact in one sentence to a fact in a distant sentence, through the same entity, term, or logical relation the translation had to preserve.

The central design choice comes from multi-hop question answering (QA) research. There, questions that look multi-step are often solvable by shortcuts, so each question must be filtered to guarantee it cannot be answered from either sentence alone. Under that filter, a translation that keeps every individual fact but severs the link between them fails the composed question and passes both single-step questions. That pattern gives both a score and a diagnosis. It is also exactly the class of error that dense single-fact QA, sentence-level learned metrics, and fluency-biased LLM judges are structurally unable to see.

Core Claim

The new signal is link preservation. We score each question twice: once composed, once as its two component hops, and count the cases where a translation passes both hops but fails the composition. This "link-break rate" isolates discourse errors (broken coreference, inconsistent terminology, lost connectives, dropped bridge sentences) from ordinary factual errors. We claim it will track expert human judgments of discourse quality better than single-hop QA with the same question budget.

The claim is falsifiable in a clean way. On controlled perturbations that break links while leaving every local fact intact, single-hop QA should stay flat and the proposed metric should drop. If dense single-hop QA catches these perturbations equally well, the multi-hop ingredient is not needed and the idea fails.

Relation to Prior Work

The motivation comes from the document-level evaluation literature:

QA-based evaluation needs no sentence alignment and directly punishes omission, which answers both failures.

The closest existing work is TREQA, Do LLMs Understand Your Translations? Evaluating Paragraph-level MT with Question Answering. It generates questions from the source or reference, answers them from the candidate, and compares answers with a learned similarity metric. We reuse two of its pieces: answer regeneration from the source, and candidate-aware generation. We differ in three ways:

  • Its questions target facts within a single paragraph.
  • It applies no shortcut filtering.
  • It reports degraded QA on non-English text, which constrains which language directions we can trust.

The method for making multi-hop questions honest comes from MuSiQue. MuSiQue composes multi-hop questions from single-hop ones and filters out those solvable by "disconnected reasoning." It showed single-hop models lose 30 F1 on its data relative to HotpotQA. We borrow that construction and apply it within a single source document rather than across Wikipedia articles, with bridges chosen specifically from discourse devices.

Two generation systems already carry multi-hop or memory ideas into translation, but neither evaluates discourse preservation:

Both are natural test subjects for our metric, not competitors to it.

Research Program

Question construction. Construction runs on the source document, or on the English reference for into-English directions. An LLM first annotates coreference chains, repeated technical terms, and explicit inter-sentence discourse relations. We then pick pairs of sentences A and B, separated by at least $d$ sentences, joined by a bridge. The bridge is one of three kinds:

  • an entity mentioned under different surface forms (for example a name in A, a pronoun in B);
  • a term that must be rendered consistently in both sentences;
  • a relation such as cause or contrast signaled by a connective.

For each pair we generate:

  • hop one, whose answer is the bridge, taken from A;
  • hop two, which asks about the bridge's property in B;
  • a composed question that never names the bridge.

Answers are regenerated from the source, following TREQA.

Filtering. A candidate question is kept only if it passes four checks:

  • it is answerable from the full source;
  • it is unanswerable when A is masked;
  • it is unanswerable when B is masked;
  • it is unanswerable closed-book.

These four checks are the MuSiQue disconnection criterion adapted to a single document.

Scoring. A fixed QA reader answers every question from the candidate translation. Answers are compared to gold answers with an LLM matcher, validated against LERC on a subset. The document score is the composed-question accuracy divided by the same reader's accuracy on the reference translation. This normalization separates reader failures from translation failures, which matters because TREQA reports that the reader weakens on non-English text.

We also report the link-break rate: the fraction of questions where both hops succeed and the composition fails. Each such failure points to a specific sentence pair, which gives the error attribution that black-box judges lack.

Language directions. The main study uses into-English directions, where the reader is strongest. English-to-X directions are a secondary analysis, and there we report the reference-normalized ceiling explicitly.

Baselines

Naive reference points. Two naive references anchor the comparison:

  • document-averaged BLEU and chrF++, the conventional segment-averaging practice described in the WMT20 Metrics Shared Task;
  • a length-ratio omission detector, since some of what we catch may simply be missing text.

The proposal must beat the length ratio on perturbations that preserve length.

Learned metrics. The strongest learned-metric competitor is xCOMET, which tags error spans within segments, alongside d-COMET. Both run on the same outputs with released checkpoints. On DiscoX, XCOMET-QE reached 34.7% consistency with human experts, as reported in DiscoX.

Strongest existing document judge. This is Metric-S, the reference-free multi-agent rubric judge from the same DiscoX paper, which reports 70.3% consistency with human experts. Beating it on DiscoX consistency would be the headline result. Matching it while adding verifiable, sentence-pair-level attribution would still be useful, but is a weaker claim. We run Metric-S with the same backbone LLM we use for question generation, so model strength is not a confound.

Direct LLM judge with chain of thought. We also include a direct LLM judge that must list error spans before scoring, the multi-granular chain-of-thought design the survey recommends. It gets a token budget equal to our full generation-plus-QA cost.

TREQA. TREQA is the most important comparison. We run it candidate-aware, with question count matched to ours. No reported result was found on Lacuna for TREQA's numeric correlations on LITERARY or PAR3HUMAN, so we re-run it ourselves.

Ablations. Two ablations isolate the key ingredient:

  • composed questions without the disconnection filter;
  • a dense single-hop variant that asks our own hop-one and hop-two questions separately.

The dense single-hop variant uses the same sentences and same reader. It removes only the composition.

Why not just…?

Why not just ask many more single-hop questions covering every sentence? Dense single-hop coverage does catch omissions, and it is the most serious threat to the proposal. It cannot catch a translation that keeps both facts but attaches the pronoun in B to the wrong character, because each single-hop question can be phrased and answered locally. The dense single-hop ablation above, run on link-only perturbations, settles this directly. If it detects them as well as composed questions do, we will say the multi-hop ingredient is unnecessary.

Why not just prompt a strong LLM judge to check coreference and terminology explicitly? This is plausible, and it is the second real threat. The survey's conciseness-bias stress test (8.5 for a translation missing about 40% of content, versus 8.0 for a complete one) shows that holistic judges are swayed by fluency. It is less clear whether a targeted checklist prompt is swayed the same way. We compare against exactly this checklist judge at matched compute. We report detection separately for bridge-sentence deletion, which is where fluency bias should hurt the judge most.

Why not just use contrastive test sets in the style of Bawden et al.? Those sets score a model's preference between fixed translations. They cannot grade an open-ended LLM output that restructures sentences, and they span only two sentences. Our questions apply to any output, and we vary the distance $d$ explicitly.

Why not just use DelTA-style string consistency? LTCR-1 checks whether a proper noun is rendered identically. It says nothing about pronouns, connectives, or synonym-based references. It would also flag a legitimate paraphrase as an error. We report it alongside our metric on Guofeng to show where the two agree and disagree.

Proposed Experiments

Controlled perturbation study. We start from reference or high-quality translations in the LITERARY data (the into-English, paragraph-level set used by TREQA), Guofeng Webnovel Chinese–English, and IWSLT2017. We inject five error types:

  • pronoun or gender swaps pointing to the wrong antecedent;
  • terminology alternation;
  • connective flips or deletions;
  • deletion of a bridge sentence;
  • deletion of a non-bridge sentence of equal length, as a control.

The decisive check is the first three types. There, single-hop and dense single-hop QA should change little, and composed-question accuracy and link-break rate should drop sharply. The effect should grow with $d$, where xCOMET, being segment-local, should be flat.

Correlation with human judgments. We measure agreement with human judgments on three sets:

  • DiscoX expert consistency, the same protocol as the reported 70.3% and 34.7%;
  • PAR3HUMAN pairwise preferences;
  • LITERARY, focusing on MQM spans labeled as discourse or accuracy errors that cross sentence boundaries.

Application study. We re-score systems whose discourse claims were previously validated with BLEU or COMET: a sentence-level baseline, sliding-window context, Doc2Doc long-context prompting, DelTA, and a thinking-enhanced model. The question is whether our metric reorders them. We specifically test whether the omission failures DiscoX attributes to thinking models show up as bridge-hop failures.

Reliability checks. We measure run-to-run variance across question-generator seeds. We also compare results with different reader families, to catch any tendency of the reader to favor translations from its own model family.