Weighting Copy Supervision by What the Extractor Can Actually See

Many systems that read text in images answer questions by mixing two output modes. Upstream, an optical character recognition (OCR) system, the component that turns pixels into words, reads the scene text. Downstream, a decoder either copies one of those recognized words or generates a word from a fixed vocabulary. Whether copying can ever be correct depends on the upstream reader. If OCR misread "PEPSI" as "PEPS1", the right answer is not among the tokens the decoder can point to. Standard training still asks the model to produce the right answer. The loss then pushes it to memorize the answer in its vocabulary or to point at the wrong token, and both lessons hurt at test time.

We propose to split the decoder's training signal by what the upstream extractor makes reachable. Each answer token falls into one of two groups: the target appears among the extracted source tokens and copying is feasible, or it does not and only generation can succeed. The two groups are then treated as two tasks whose weights are rebalanced during training from cheap loss statistics. The claim is deliberately modest about the bottleneck. No training trick can recover a word the OCR never produced. Static training wastes part of what the extractor does provide and teaches the copy switch the wrong statistics. Extractor-aware dynamic weighting should close that gap and degrade more gracefully when the extractor gets worse or the domain shifts.

Core Claim

A pointer-generator is a decoder that at each step mixes a vocabulary distribution with an attention-based "copy" distribution over source tokens. A learned switch $p_{gen}$ controls the mix (Get To The Point). Two failure modes of this switch are documented in the file, and they pull in opposite directions. See et al. found the switch collapses toward verbatim extraction. Pointer-Generator Networks for Low-Resource Machine Translation: Don't Copy That! found it copies meaningless subwords when the source and target tokenizations do not line up.

Our reading is that both are symptoms of the same thing: the training signal never tells the switch when copying was even possible given the upstream representation. In scene-text question answering, M4C (Iterative Answer Prediction With Pointer-Augmented Multimodal Transformers for TextVQA) and SSB (Simple is not Easy) both state that accuracy is gated by OCR quality. Both still train with a single statically weighted objective. The new element is to define the sub-tasks of multi-task weighting by extractor reachability, not by output head or dataset. With this split, the weights automatically track how good the extractor is.

Concretely, let $\mathcal{R}$ be the set of decoding steps whose target token matches at least one extracted source token. Let $\mathcal{U}$ be the remaining steps, where only generation can be correct. Let $L_\mathcal{R}$ and $L_\mathcal{U}$ be the mean negative log-likelihoods of the usual mixture distribution over each set. We train on

$$L(t) = w_\mathcal{R}(t)\, L_\mathcal{R} + w_\mathcal{U}(t)\, L_\mathcal{U},$$

where $t$ is the training step. The weights $w_\mathcal{R}$ and $w_\mathcal{U}$ are set from scalar loss trajectories rather than per-task gradients. They combine the inverse moving-average scaling of Lakkapragada et al. with a validation-based brake taken from CoBa. That brake lowers a group's weight once its held-out loss starts rising. The brake matters most for $\mathcal{U}$. Rising held-out loss on generation-only steps is the signature of the model memorizing answers that OCR missed, which shows up at test time as confident hallucination.

Research Program

The first component is measurement. For each benchmark and OCR source we compute an extractor ceiling: the fraction of answers producible by some combination of vocabulary tokens and exactly matching OCR tokens. We report every model as achieved accuracy alongside this ceiling. This turns the vague phrase "gated by OCR" into a number, and it separates two claims that prior work blurs. The first is gains that come from better use of available signal. The second is gains that would require recovering discarded signal, which we do not claim. The framing follows the information-loss argument in Lavrenko et al.. It also follows the finding of Tuia et al. that richer fusion does not protect against shift when the backbone fails. Both papers act as constraints here: they tell us not to promise ceiling-breaking.

The second component is the training method above, implemented on top of an M4C-style architecture with a frozen OCR front end. M4C supplies the decoder. It copies over OCR tokens through a dynamic pointer and generates the rest from a fixed vocabulary. The split into $\mathcal{R}$ and $\mathcal{U}$ is computed once per example from the OCR output, so the added cost is a handful of scalar moving averages per step.

We choose loss-scalar weighting over gradient-based balancing such as GradNorm for two reasons. The two groups share the whole network, so per-group gradient norms on shared layers double the backward cost. And CoBa reports roughly 1.5x faster epochs than gradient-based balancers. GradNorm is still run as a comparison.

The third component is diagnostic. We measure how well calibrated the copy switch is: whether $1-p_{gen}$ predicts membership in $\mathcal{R}$ on held-out data. We also count two error types, wrong-generate-when-copyable and wrong-copy-when-uncopyable. If the method works, both should fall, and the switch should track reachability more closely than under static training.

Baselines

The naive reference point is the published M4C recipe: one statically weighted objective over the combined vocabulary and OCR prediction space. This is exactly what our method modifies. It reports 39.01% accuracy on TextVQA, 0.462 ANLS (average normalized Levenshtein similarity, a character-tolerant answer score) on ST-VQA, and 63.9% on OCR-VQA (M4C). We retrain it ourselves with the same OCR tokens, features, schedule and seeds as every other arm, so that the comparison isolates the loss.

The strongest existing method on these benchmarks in the file is SSB (Simple is not Easy). It splits OCR tokens into visual and linguistic streams and adds an ANLS policy-gradient term that already partially tolerates OCR misspellings. It reports 45.66% on TextVQA and CIDEr 89.63 on TextCaps. No ST-VQA ANLS number for SSB was found on Lacuna. SSB is the right competitor because its policy-gradient loss is an alternative answer to the same OCR-noise problem. We apply our reweighting on top of SSB as well as on M4C, giving both the same tuning budget. Beating it means an accuracy gain over our reproduced SSB that exceeds seed variance, together with a smaller fraction of the extractor ceiling left unused. The second criterion matters as much as the first.

Two balancing baselines receive the same $\mathcal{R}/\mathcal{U}$ split. One is a static weight on $L_\mathcal{U}$ chosen by grid search with a compute budget matched to our total training cost. The other is GradNorm on the same two groups. Together they test whether dynamics matter and whether cheap scalar weighting suffices.

The key ablation keeps the dynamic weighting but splits steps into two random groups matched in size to $\mathcal{R}$ and $\mathcal{U}$. If this random-split control performs as well, the gain comes from generic reweighting noise and not from extractor-aware grouping, and the core claim fails.

Why not just…?

Why not just use a better OCR system? This is the real threat, and we expect it to win in absolute terms; the bottleneck literature says it should. The proposal is not a substitute. Its claim is complementarity and graceful degradation. We run every arm with two front ends of different quality and with synthetic OCR corruption. The method is useful only if it closes a constant or larger share of the ceiling gap at every extractor quality, and if its advantage grows as OCR degrades. If its gains vanish with the stronger OCR, we will report that as a negative result.

Why not just drop the uncopyable examples, or tune one static weight? Dropping $\mathcal{U}$ throws away real generation-only answers, such as yes/no responses and counts. The grid-searched static weight is run directly. GradNorm reports finding a balance in one run that beat a large manual grid (GradNorm), and CoBa argues that the right weight changes over training. But neither result is on copy mechanisms, so the matched-budget grid is a genuine test, not a strawman. If it ties, we conclude that the grouping, not the dynamics, is the contribution.

Why not just rely on SSB's ANLS policy-gradient loss? That loss rewards near-miss strings. It helps when OCR is slightly wrong but gives no signal when the answer is entirely absent. It also does not change how the copy switch is trained. Stacking our weighting on SSB measures whether the two overlap. Error analysis split by edit distance between OCR and gold answer will show where each one helps.

Why not just use a larger multimodal model or deeper fusion? The file's evidence argues against this as a fix for discarded signal. Li et al. found question-guided convolutions cannot recover what a fixed backbone lost. Tuia et al. found tensor fusion gives no protection under geographic shift. SSB beat more complex models with simpler fusion. We do not test giant models; the ceiling metric is designed so that such comparisons could be added later on fair terms.

Proposed Experiments

The main runs train on TextVQA and evaluate on TextVQA, ST-VQA and OCR-VQA, plus TextCaps for the generation-heavy setting. DocVQA (DocVQA) serves as a dense-text stress test where the reachable fraction differs sharply. The extractive BERT baseline there reports 0.665 ANLS; no reported M4C number on DocVQA was found on Lacuna. Every result is reported with its extractor ceiling, three seeds, and the switch-calibration and error-type diagnostics.

The first decisive check is the corruption sweep. We inject character substitutions and token deletions into the OCR output at increasing rates, at test time only and at both train and test time. This tests whether the method's accuracy-versus-noise curve is flatter than static training's and SSB's.

The second check is the random-split control described above.

The third is a domain-shift check: train on TextVQA and test zero-shot on ST-VQA. This tests whether CoBa's validation brake on $\mathcal{U}$ reduces confident vocabulary answers on questions whose answers were copyable.

A secondary transfer test applies the same reachability split to text-only copying. On CNN/Daily Mail, the pointer-generator with coverage reports ROUGE-1 39.53 (Get To The Point), and the question is whether reweighting reduces extractive collapse. On the low-resource translation pairs of the Don't Copy That! study, no exact BLEU numbers were found on Lacuna, and the question is whether defining reachability by subword overlap stops random subword copying. A positive result there would suggest the mechanism is general. A null result would confine the claim to settings where the extractor's outputs are discrete, inspectable tokens.