Knowing Whether You Misread or Mistranslated: Selective Document Image Translation for Low-Resource Scripts

Suppose a system is asked to translate a photographed health leaflet or invoice written in Sinhala, Georgian or Khmer. It usually produces fluent English whether it read the page correctly or not. In these scripts the most common failure is not a bad translation of the right words. It is a confident translation of the wrong words: the model misreads a stacked character, scrambles the reading order of a table, or silently copies text it could not parse. We propose a translation model that can decline to translate individual segments of a page, and that says why it declined. Its abstention signal is split into two parts. One gate asks whether the model read this segment correctly. The other asks whether the model translated what it read faithfully. The model outputs the segments it trusts and flags the rest, with the flag naming the likely cause.

The bet is that one overall confidence score cannot separate these two failures. A misreading followed by a fluent translation looks confident, so the model answers when it should abstain. A hard but correctly read segment looks uncertain, so the model abstains when it should answer. Splitting the signal along the image-to-transcription-to-translation chain should therefore let the system answer more segments at the same error rate. It also gives a signal that can clean noisy training pairs in languages where standard filters cannot be built. Both claims can be tested, and both could fail.

Core Claim

Recent document image translation models emit a source transcription before the translation. For these models, a selective predictor (a model paired with a rule that decides, per input, whether to answer or abstain) should use two gates. The reading gate is read from the model's internal state during the transcription phase. The translation gate is an adequacy score that runs both directions. We claim this pair beats thresholding the translation's own sequence probability on segment-level risk-coverage curves. We expect the gain to concentrate in low-resource scripts, where reading errors dominate, and to shrink toward zero on high-resource English–Chinese documents. The prediction is falsifiable in both directions.

This combination is new relative to four lines of work in the file. The document image translation models all translate every input unconditionally. LayoutDIT reorders words, segments sentences and then translates with a layout-aware encoder. ZoomDIT reassembles "chaotic OCR" word fragments into sentences before translating. SSR makes a multimodal LLM transcribe the page as Markdown before translating, which keeps its OCR ability from collapsing. None of the three can abstain, and all three were tested only on English–Chinese.

Selective prediction work supplies the machinery but not the setting. The Art of Abstention adds an error-regularization loss that pushes confidence on historically wrong examples below confidence on correct ones. It stops at classification and names generation as open. ReCoVERR is our main motivation. It shows that vision-language models abstain too often because reasoning uncertainty hides perceptual competence, and it fixes this by verifying perceptual sub-answers. We move that insight into translation and replace its external LLM and entailment-model calls with signals the translator already computes. OpenWHO supplies professionally translated low-resource health documents, but it strips the layout away and never abstains.

Research Program

Base model. The translator follows SSR's recipe: Qwen2-VL fine-tuned to go from image to source Markdown to target. We choose SSR because its intermediate transcription is the natural point at which to split the abstention decision. Its reported sample efficiency also matters in low-resource settings: trained on 500 samples, it beat standard fine-tuning trained on 100,000 samples in cross-domain tests.

Reading gate. This is a small classifier (a linear probe) on intermediate activations taken while the model emits each segment's transcription. Can VLMs Recall Factual Associations From Visual References? showed that such probes can detect a failure to bind visual input before the model decodes text. We supervise the probe with segment-level character error against the gold source. We add Xin et al.'s error-regularization term so that confidence ranks segments by how often they were misread during training.

Translation gate. This gate reuses the adequacy score from Dual Conditional Cross-Entropy Filtering at inference time. Let $s$ be the model's own transcription of a segment and $t$ its translation. Let $H_{\to}(t\mid s)$ be the length-normalized cross-entropy of the translation given the transcription, and let $H_{\leftarrow}(s\mid t)$ be the cross-entropy of the transcription given the translation. The reverse term costs one extra teacher-forced pass, meaning the model scores a fixed sequence rather than generating a new one. The gate score is

$$a(s,t)=\exp\!\Big(-\big|H_{\to}(t\mid s)-H_{\leftarrow}(s\mid t)\big|-\tfrac12\big(H_{\to}(t\mid s)+H_{\leftarrow}(s\mid t)\big)\Big).$$

The original method needs separately trained forward and backward "helper" models built from clean parallel data. Low-resource languages do not have that data, which is the main open problem the file identifies for this filter. Here the same multilingual translator scores both directions, so no clean seed corpus is needed. The cost is that the scorer is only as good as the translator, and the experiments must check whether that circularity hurts.

Combining the gates. A segment is emitted only if the reading score clears threshold $\tau_r$ and the adequacy score clears $\tau_t$. Both thresholds are chosen on held-out data so that selective risk stays below a target $\rho$ while coverage is as high as possible. Following Geifman and El-Yaniv, selective risk is the error rate on answered segments and coverage is the fraction of segments answered. A segment is counted as an error when its ChrF against the reference falls below a fixed threshold. A human critical-error judgment is used on the real-scan subset. When a segment is rejected, the output records which gate fired. A reading failure routes to rescanning or human transcription, and a translation failure routes to a translator.

Data. No parallel, layout-annotated low-resource document corpus exists, so we build one. We render OpenWHO documents in Sinhala, Georgian, Armenian, Kazakh, Macedonian, Tetun and Somali into multi-column, table and form layouts. We then apply scan distortions: folds, blur, stamps and uneven lighting. For a real-world test we commission English translations of segments from KH-FUNSD, a set of 158 Khmer business documents on which even LayoutLMv3 reaches only 68.6 F1 for entity classification. That result signals that reading Khmer is itself the bottleneck. DITrans (English–Chinese) serves as the high-resource control.

Filtering use. The same two gates, run over training pairs, give a filter that needs no clean seed data. We test it by injecting noise into the rendered training data. The noise types come from Khayrallah and Koehn: copied source, misalignment and wrong language. We add one layout-specific type, shuffled reading order, which 1D toolkits such as OpusFilter cannot see.

Baselines

The naive reference point is the SSR-style model with no abstention, which answers everything. Its risk at full coverage anchors every curve. The strongest existing selection method is softmax response for sequences: threshold the length-normalized log-probability of the whole transcription-plus-translation output, as entropy thresholds were used in EHRSQL. We also train it with Xin et al.'s error regularization, since they report roughly a 10% reduction in the area under the risk-coverage curve on GLUE from that loss.

The second strong competitor is a cascade. It runs a separate OCR engine and uses its character confidence as the reading gate. It then applies text-only translation to the OCR output and uses the same adequacy score as the translation gate. The cascade is the most direct test of whether an end-to-end model is needed at all. LayoutDIT reports that cascaded OCR-then-MT pipelines reach 30–37 BLEU on ReadingBank against its own 48.20. The cascade may still select well even if it translates worse.

The third competitor is Monte Carlo dropout agreement over $k$ samples. Its compute is matched by allowing it the same number of extra forward passes our method uses, and a larger-$k$ version is reported separately.

Two ablations isolate the key ingredient. The first keeps only the reading gate and the second keeps only the translation gate. A third ablation uses a single probe on translation-phase activations, which tests whether the gain comes from the split or from probing.

All methods use the same fine-tuned base model and the same training data. Each gets the same budget for tuning its selection hyperparameters on a shared calibration split. For the strongest baseline, SSR's reported quality is 57.23 BLEU on in-domain DoTA and 33.61 BLEU cross-domain. We found no reported result on Lacuna for selective or segment-level abstention in document image translation in any language. Our method beats a baseline if, at matched selective risk of 10%, it answers at least five coverage points more segments than sequence softmax response on the low-resource test sets. It must also do no worse on DITrans and add no more than one extra teacher-forced pass per segment.

Why not just…?

Why not just threshold the translation's own sequence probability? This is the real threat, and it is the main baseline. Our reason to doubt it comes from Prediction Difference Regularization, which finds that translation models often give higher confidence to perturbed tokens than to valid references. A fluent translation of a misread transcription should therefore look confident. If sequence confidence matches the split gates on the Sinhala, Georgian and Khmer sets, the split adds nothing and we will report that. The per-gate diagnostic (below) will show directly whether sequence confidence ranks misread segments above correctly read ones.

Why not use a pipeline with an OCR engine's own confidence? This baseline decomposes the decision for free, and it is also a serious threat. Standard OCR engines fail on Khmer's unspaced, stacked script, as KH-FUNSD documents. Their confidence scores are not calibrated for scripts they were not built for. Running the cascade as a baseline on the same pages settles the question empirically. If it matches our risk-coverage curve, the practical recommendation becomes "cascade for selection, end-to-end for quality," which is still a useful finding.

Why not verify uncertain outputs the way ReCoVERR does? ReCoVERR recovers up to 20% coverage on VQAv2 and A-OKVQA. It does so with GPT-3.5 sub-questions and an entailment model, and neither exists at useful quality for Tetun or Somali. It is also too slow for batch document work. We borrow its diagnosis, that perception is reliable where reasoning is not, and implement the check with the translator's own activations and a scoring pass.

Why not just clean the training data so abstention is unnecessary? Filtering and abstention are complementary, not substitutes. The WMT 2019 low-resource filtering task shows how low the ceiling is even with good filtering: Nepali and Sinhala NMT peaked at 6.9 and 6.4 BLEU. Residual errors at that level will reach high-stakes users unless something flags them. We still test filtering head-on in the noise-injection experiment.

Proposed Experiments

The first decisive check is segment-level risk-coverage on the rendered OpenWHO languages and on real KH-FUNSD scans. We report the area under the curve and coverage at 5% and 10% risk, and compare against every baseline above.

The second check tests the mechanism, and it predicts where the method should and should not help. Among segments that the no-abstention model gets wrong, we measure the fraction whose transcription error exceeds a threshold. The gain over sequence confidence should track that fraction across languages. It should be large for Khmer and Sinhala and near zero for DITrans. If the gain does not track the reading-error share, the mechanism we propose is wrong even if the aggregate numbers look good.

The third check measures attribution accuracy on rejected segments. When the reading gate fires, transcription error should be high. When only the translation gate fires, transcription should be correct and ChrF low.

The fourth check is the noise-injection filtering experiment. We compare four training sets: unfiltered data; data filtered by LaBSE similarity with language identification and length ratio; data filtered by dual cross-entropy scoring with helpers trained on the scarce clean data; and data filtered by our gates. We sweep 5–50% of each noise type. Copied-source noise is the case to watch, since it cost 9.6 BLEU at only 5% in Khayrallah and Koehn's setting. Shuffled reading order is the case 1D filters should miss. Success means our filter recovers most of the clean-data ChrF under copied-source and reading-order noise, where the 1D filters do not.

The final check is document-level context. OpenWHO found that smaller models sometimes degrade when given whole-document context. We therefore report whether whole-page context helps or hurts the reading and translation gates separately at 2B and 7B model scale.