When a machine translates a Chinese web novel into English, many of its worst errors are wrong answers to questions about the story, not bad sentences. Whether a character is "he" or "she," whether a speaker addresses someone as "senior brother" or "master," and whether two nicknames name the same person all depend on facts established chapters earlier. Some of those facts can only be recovered by chaining two or three of them together: the woman in the green robe is the sect leader's daughter, the sect leader is the protagonist's teacher, so she is his senior and he speaks to her with deference. We propose to make these hidden questions explicit. Before translating each chapter, the system writes down the story facts its translation choices depend on. It answers each one by retrieving evidence step by step from earlier source chapters and records the answer, together with its evidence chain, in a ledger. Human editors then review only the uncertain links in those chains, not the translated text. A single human fix to one link automatically corrects every sentence, past or future, whose translation depended on it.
The opportunity is that human effort in machine-assisted translation is spent at the wrong level. Post-editors read target sentences one at a time and cannot see the long-range facts that make a pronoun or honorific wrong. Checking a short claim such as "Lin Yue and the Green-Robed Maiden are the same person (ch. 3, ch. 17)" takes seconds, and the answer governs hundreds of sentences. If most discourse errors in long literary translation trace back to a small set of misresolved multi-step story facts, then reviewing those facts should buy far more quality per human minute than reviewing text.
Core Claim
Our central hypothesis is this: at a fixed budget of human minutes, having bilingual editors verify and correct the uncertain steps in automatically built chains of story facts yields fewer discourse-level translation errors than spending the same minutes post-editing the target text. It also beats routing a quality-estimated subset of sentences to editors. The gain should come from two things. The first is fan-out, meaning one corrected fact propagates to many dependent translation decisions, including decisions in chapters already translated. The second is the fact that the errors worth catching sit at multi-hop links that neither long-context models nor one-shot retrieval resolve reliably. The second part is an empirical bet that we test first, and the proposal changes shape if it fails.
What Prior Work Establishes and Leaves Open
Three findings motivate the proposal. NovelHopQA built 4,000 multi-hop questions over 64k–128k-token English novels. It found that accuracy falls about 12 points from 1-hop to 4-hop questions at 64k, that the strongest models dip below 80% on 4-hop questions at 128k, and that retrieval-augmented models sit near 50%, 25–35 points below golden context. Its failure categories, entity confusion and missing final-hop integration, are exactly the failures that produce wrong pronouns and honorifics in translation. NovelHopQA only diagnoses English reading comprehension, however. It never asks whether these failures surface in generated translations. The WMT 2023 discourse-level literary translation task released the GuoFeng web-novel corpus and showed that document context improves gender and anaphora consistency, but its humans only score outputs after the fact. TRANSAGENTS translates 240 chapters with simulated editor and translator agents that share a static glossary and book summary. It achieves high model-judged quality with low d-BLEU and no real humans, and its static notes cannot resolve a fact that only becomes determinable by combining chapters.
Two lines of work supply the method. Interleaved retrieval, where each reasoning sentence becomes the next search query (IRCoT) or each draft seeds the next retrieval (ITER-RETGEN), gives us an API-compatible way to answer ledger questions over earlier source chapters. We avoid DRAGIN's attention-based triggering because it needs internal model states, which proprietary translation models do not expose. Joshi et al. showed that human corrections of intermediate multi-hop steps improve reasoning, and their ablation found that free-form corrections helped far more than error-category labels. We therefore ask editors for corrected facts, not labels. Their setting was short English QA chains, where a correction fixes one answer. Ours is a long bilingual document, where a correction fixes many decisions.
One finding constrains the design. GRAT shows that linear retrieval chains collapse when a single step fails, and it answers with tree search. We treat human verification of low-confidence links as a cheaper form of backtracking. DiffResearch identifies the inability of left-to-right generation to revise earlier sections once new evidence arrives. The ledger addresses this directly by making earlier chapters revisable when a fact is corrected.
Research Program
The system has three parts. The first is a question writer. For each source chapter it lists the translation-critical decisions the chapter contains, such as the referent and gender behind each ambiguous pronoun, the relationship behind each form of address, the canonical rendering of each alias, and the identity of recurring objects and techniques. It phrases each decision as a question about the story. The second is a chain resolver. It answers each question by IRCoT-style interleaved retrieval over the Chinese source of all earlier chapters and records the answer, the chain of supporting passages, and a confidence estimate from agreement across several sampled chains. The third is the translator, which translates the chapter conditioned on the resolved ledger entries. It also records which target sentences depend on which entries, so that any corrected entry triggers retranslation of exactly those sentences anywhere in the book.
Humans enter through a review queue. Each item is one link of one chain, shown as a claim with its two or three source passages, and the editor accepts it or writes the corrected fact. Items are ordered by expected errors prevented per minute of review:
$$\text{priority}(e) = \frac{(1 - c_e)\, d_e}{\tau_e}$$
Here $c_e$ is the resolver's confidence in link $e$, $d_e$ is the number of translation decisions downstream of $e$ in the dependency record, and $\tau_e$ is the predicted review time, estimated from passage length. This form assumes that errors prevented scale with the probability the link is wrong times its fan-out. We will check that assumption by comparing it against confidence-only ordering.
Stage zero is an error audit, and it decides whether the multi-hop part of the proposal is needed. On a sample of GuoFeng chapters translated by a strong long-context model, bilingual annotators mark discourse errors in the style of Multidimensional Quality Metrics (MQM), the human error-annotation scheme used at WMT. For each error they label how many cross-chapter facts must be combined to make the right choice. If errors needing two or more hops are rare, a flat ledger of single facts reviewed by humans is the right system, and we will build that instead.
Baselines
The naive reference point is chapter-by-chapter translation by the same backbone model with the preceding chapter in context and no human involvement. It anchors how much any intervention helps.
The strongest automated competitor is TRANSAGENTS itself, run on the same novels. The TRANSAGENTS paper reports a d-BLEU of 25.0±2.4 and a GEMBA-DA of 87.7±0.2, against 85.8±0.5 for GPT-4-Turbo (TRANSAGENTS). It is the strongest competitor because it already attacks book-level consistency with a glossary and summary. We will rerun it with our backbone model so that differences reflect design, not model choice, and we will compare on the targeted decision metric described below, not only on its own reported scores. We found no reported number on Lacuna for the top WMT 2023 systems, a fine-tuned mBART and a prompted GPT-3.5 system.
The strongest human-in-the-loop competitor is conventional post-editing at a matched budget. Professional bilingual editors receive the same number of minutes per chapter as our ledger reviewers and post-edit the baseline translation in document view. A second variant routes them to the sentences an automatic quality estimator flags as worst. No reported result was found on Lacuna for quality-estimation-routed literary post-editing, so we measure it ourselves. Editor pools are disjoint across conditions, and chapter assignment is balanced across conditions.
Three ablations isolate the key ingredient. The first is the full ledger with no human review, which is effectively IRCoT applied to translation and shows what automation alone achieves. The second is a ledger restricted to single-hop lookups, which tests whether multi-hop chaining matters. The third orders human review by confidence alone, without fan-out, which tests whether prioritizing by fan-out matters. Beating a baseline means a statistically significant reduction in wrong translation-critical decisions and in MQM major and critical discourse errors, with no loss in GEMBA-DA or human preference.
Why not just…?
Why not just give a long-context model the whole novel? NovelHopQA directly measures this on English novels, and hop depth, not length, drives failures, with the best models below 80% on 4-hop questions at 128k. We cannot assume the same holds for translation choices, so the long-context model with the full preceding source serves as our naive baseline and is also the system audited in stage zero.
Why not just build a glossary and character sheet up front, as TRANSAGENTS does? This is the most serious threat. If most translation-critical facts are fixed attributes known by chapter one, a static sheet reviewed once by a human would capture nearly all of our gain at lower cost. Relationships and identities in serialized fiction change and are revealed late, so we expect it to fail on facts that emerge over time. The stage-zero audit and the single-hop ablation settle this. We will also run a condition in which humans review a TRANSAGENTS-style sheet at the same budget.
Why not just route the lowest-quality sentences to post-editors? Quality estimators score sentences, and a sentence with a wrong honorific is fluent and locally plausible. The WMT 2023 findings note that sentence-level automatic metrics missed discourse errors. Whether this holds for modern estimators is an empirical question, and the matched-budget routing baseline answers it.
Why not just let the automated resolver fix everything, with no humans? Linear chains break at single weak links (GRAT), and 50% retrieval accuracy on NovelHopQA suggests the resolver will be wrong often. If the no-human ledger closes most of the gap to the human-reviewed ledger, the human component is not worth its cost, and we will report that.
Proposed Experiments
We will use the TRANSAGENTS test novels drawn from the GuoFeng corpus, 20 novels and 240 chapters, plus additional GuoFeng novels in contemporary genres, where TRANSAGENTS was weakest. On held-out chapters, bilingual annotators build a translation-decision test set by marking every translation-critical decision, its gold choice, and the hop depth of its supporting facts. The primary metric is decision accuracy, stratified by hop depth. Secondary metrics are MQM discourse errors, d-BLEU, GEMBA-DA, and pairwise human preference. Cost is measured in human minutes and model calls.
Three checks decide the outcome. The first is the stage-zero audit: if errors needing two or more hops are not a substantial fraction of discourse errors, the multi-hop resolver is dropped. The second is the curve of decision accuracy against human minutes, where ledger review must dominate text post-editing and routed post-editing across budgets, not only at one point. The third is a fan-out analysis measuring how many target sentences each accepted human correction changed, and whether the changes to already-translated chapters were improvements. This last check tests the claim that correcting a fact is worth more than correcting a sentence.
References
- NovelHopQA: Diagnosing Multi-Hop Reasoning Failures in Long Narrative Contexts
- Findings of the WMT 2023 Shared Task on Discourse-Level Literary Translation: A Fresh Orb in the Cosmos of LLMs
- (Perhaps) Beyond Human Translation: Harnessing Multi-Agent Collaboration for Translating Ultra-Long Literary Texts
- Interleaving Retrieval with Chain-of-Thought Reasoning for Knowledge-Intensive Multi-Step Questions
- Enhancing Retrieval-Augmented Large Language Models with Iterative Retrieval-Generation Synergy
- DRAGIN: Dynamic Retrieval Augmented Generation based on the Real-time Information Needs of Large Language Models
- Improving Multi-Hop Reasoning in LLMs by Learning from Rich Human Feedback
- GRAT: Guiding Retrieval-Augmented Reasoning through Process Rewards Tree Search
- DiffResearch: Harnessing Diffusion Language Models for Automated Literature Review