Degradation-Aware Trust Between Pixels and Language for Low-Resolution Text

When text in a photo is small or blurred, single characters become truly ambiguous: a thin vertical stroke could be "1", "l", or "I". Recent systems resolve this by consulting context, usually a language model that knows which strings are plausible words. The same move breaks on text that is not a word, such as license plates, serial codes, times, and prices. There the language model "corrects" a valid rare string into a common one. We argue this is mostly a problem of misplaced trust. How much a system should rely on context ought to depend on how much visual evidence survived the blur. Models trained on synthetic, mathematically downsampled images misjudge that amount on real camera blur, so they defer to language at the wrong moments.

We propose to learn a per-character estimate of how reliable the visual evidence is, and to fit it on a small amount of real optically degraded data rather than on synthetic degradations. This estimate then sets the weight on the language prior, both in recognition and in the text prior that guides super-resolution (upscaling the image so the text becomes legible again). The claim to test is concrete. On real low-resolution text, this calibrated weighting should keep most of the accuracy gain that language priors give on dictionary words, while removing most of the over-correction and stroke hallucination they cause on non-lexical strings. Global confidence rescaling and learned modality balancing should not achieve that split.

Core Claim

Existing prior-guided super-resolution treats the recognizer's character probabilities as trustworthy guidance, or adds a language model on top. TPGSR feeds the probability sequence from a pretrained recognizer called CRNN into the upscaling network. C3-STISR adds a character-level language model that turns "blrd" into "bird" before guiding reconstruction. NCAP shows that hard categorical priors become overconfident across the gap between low- and high-resolution images, and it responds by removing categories altogether. On Calibration of Scene-Text Recognition Models shows that text recognizers are badly overconfident and that calibration helps beam search. None of these ask whether a recognizer's confidence tracks the amount of real optical degradation, or use that to decide when language should win.

Our mechanism is a weighted combination of the visual and linguistic evidence at each character position $t$:

$$\log p(c_t \mid x, c_{\neg t}) = \log \tilde p_v(c_t \mid x) + \lambda_t \,\log p_\ell(c_t \mid c_{\neg t}) + \text{const}.$$

Here $\tilde p_v$ is the recognizer's visual posterior after a degradation-conditioned calibration map, and $p_\ell$ is a bidirectional character language model that conditions on the neighbouring characters $c_{\neg t}$. The weight $\lambda_t \in [0,1]$ is the product of two learned scores. The first is an evidence-strength score, which falls as the character's visual support weakens. The second is a lexicality score, the estimated probability that the whole string is natural language rather than a code.

The calibration map and $\lambda_t$ are fit on real TextZoom low- and high-resolution pairs. Because the high-resolution crop is legible, it tells us, for each low-resolution character, whether the visual-only prediction was right. That gives direct supervision for how much the visual model should be trusted under real optics.

The sim-to-real direction does two jobs in this design. It is the reason synthetic-fit trust fails, and fitting on real data is the fix. Scene Text Image Super-Resolution in the Wild documented that models trained on bicubic downsampling fail on real lens blur. We hypothesize the same holds for confidence, which is a separate question from accuracy.

Research Program

Stage 1: measure the failure. We split the TextZoom test set into lexical strings (in a standard English word list) and non-lexical strings (containing digits, mixed case codes, or out-of-dictionary tokens). The split is fixed before any modeling. On each subset and difficulty level (Easy, Medium, Hard), we measure three things for the existing prior-guided models: accuracy, an over-correction rate, and expected calibration error (ECE, the average gap between stated confidence and actual accuracy). The over-correction rate is the fraction of strings where a visual-only reading was correct but the language-assisted output is wrong. If over-correction on non-lexical strings is small, the proposal has no problem to solve, and we would say so.

Stage 2: fit trust from synthetic data and from real data. - Synthetic-fit variant: train the calibration map and $\lambda_t$ on MJSynth/SynthText images with bicubic and Gaussian degradation. - Real-fit variant: train them on TextZoom training pairs, with a subsampled version of a few hundred to a few thousand pairs to test data efficiency. - Evaluation: compare the two on real TextZoom test data and zero-shot on SVT and SVTP.

The evidence-strength head is small. It reads the recognizer's per-position features and is trained with a binary loss to predict visual-only correctness. The lexicality score is a string-level classifier over the recognizer's top hypotheses.

Stage 3: use the fused posterior as the prior for super-resolution. We use the combined posterior as the text prior in a TPGSR-style network, with an otherwise unchanged backbone. The aim is for guidance to be confident exactly where the evidence warrants it, so the upscaler stops painting language-model strokes onto codes.

Stage 4: test a structured-document transfer (secondary). On European Accident Statements forms, we ask whether the learned lexicality score recovers part of what that paper's hand-specified field types provide, without being told the field type. That paper reports a 37-point drop in error on "Time" fields when the field type is given. The forms are not resolution-degraded, so this tests the context-gating half of the idea only.

Baselines

Naive reference points. The first is bicubic upsampling followed by CRNN. The second is TSRN, the sequence-aware super-resolution network from Scene Text Image Super-Resolution in the Wild, which TPGSR reports at 41.4% average CRNN accuracy on TextZoom. These set the floor.

Strongest existing method. This is C3-STISR. It is the only method in the file that combines recognition, visual, and linguistic clues, so it is both the most accurate and the one most exposed to over-correction. It reports 53.7% average CRNN accuracy on TextZoom and 39.8% on the Hard subset. We will retrain it with the authors' settings, the same CRNN evaluator, and the same data. Where our method uses real TextZoom pairs, C3-STISR already trains on them too, so neither side gets extra real data.

Other prior-based baselines. We include TPGSR at its reported 49.8%, and NCAP, for which the file gives only a relative "3.5% improvement" and no absolute TextZoom number, so we will reproduce it. These isolate categorical versus non-categorical priors. Text Gestalt, at a reported 48.9%, serves as the purely visual, stroke-based disambiguation reference. Every baseline gets the same hyperparameter search budget (the same number of configurations) and the same GPU budget.

What counts as beating C3-STISR. Beating it means matching its lexical-subset accuracy within one point while cutting the non-lexical over-correction rate by at least half, with a net gain in overall and Hard-subset accuracy. Winning only on average accuracy is not the claim.

Ablations that isolate the key ingredient. - Temperature-scaled fusion: replace the calibration map and $\lambda_t$ with one global temperature and one global language weight, both fit on the same real data. - Synthetic-fit versus real-fit trust: the Stage 2 comparison, which tests the sim-to-real part of the claim. - Lexicality score removed: keep evidence strength only.

Why not just…?

Why not just temperature-scale the recognizer and tune one global language weight? This is the most serious threat. On Calibration of Scene-Text Recognition Models shows that simple calibration lowers word-level ECE from 6.0% to 1.7%. A single temperature can fix average overconfidence. It cannot make the model trust language on a blurred dictionary word and distrust it on an equally blurred plate. That requires the weight to vary by string type and by local evidence. The temperature-scaled fusion ablation settles this directly. If it closes most of the non-lexical over-correction gap, the per-character mechanism is not needed, and we will report the simpler method as the result.

Why not just train with modality masking, as BUSNet does? BUSNet randomly hides vision or language during training so its decoder learns to use either one. It reports 95.5% on SVT when trained on synthetic data. The paper itself names out-of-vocabulary strings and small symbols as its main failure. Its balance is learned on synthetic degradations and never told how reliable real optics are. We will train a BUSNet-style decoder on the same data and compare over-correction on the non-lexical split. If masking alone matches us there, our contribution reduces to the real-data calibration step.

Why not just drop the language model on low-resolution text? Removing context protects codes but gives up the gains on words. The fused visual-plus-linguistic system in C3-STISR improves over the best purely visual prior at the time, 53.7% versus 48.9%. The "no language" arm of our lexical/non-lexical split measures this trade-off exactly. We expect it to win on codes and lose clearly on Hard-subset words.

Why not just train everything on real TextZoom pairs and skip the synthetic question? Accuracy models already do this. What is new is fitting trust on real data and showing that synthetic-fit trust transfers badly. If the synthetic-fit and real-fit variants perform the same, that half of the claim is wrong, and the proposal becomes a calibration-for-fusion paper.

Proposed Experiments and Decisive Checks

Primary benchmark. The primary benchmark is TextZoom, reporting CRNN, ASTER, and MORAN accuracy by difficulty level and by lexical versus non-lexical subset. The decisive check is a two-by-two result: accuracy on lexical and non-lexical subsets, comparing our method against C3-STISR and against the temperature-scaled fusion ablation. Success means our method keeps lexical accuracy and substantially raises non-lexical accuracy, while temperature scaling does not.

Hallucination check. For super-resolution, we measure the hallucination rate. This is the fraction of upscaled images whose recognized string changes from correct under the unguided TSRN to incorrect under guidance. We report it on the non-lexical subset, alongside PSNR and SSIM (pixel-level image quality scores) for completeness. We do not optimize for them, given the known mismatch between pixel scores and legibility.

Calibration check. We report per-difficulty ECE for synthetic-fit versus real-fit trust. We expect a gap concentrated on the Hard subset.

Generalization checks. - Zero-shot recognition on SVT and SVTP, compared with PlugNet's reported 92.3% on SVT and BUSNet's 95.5%. - The EAS forms experiment, testing whether the learned lexicality gate recovers part of the benefit of oracle field typing. That paper reports 22.8% CER with field typing and 51.8% without it.

Negative result that would stop the work. If Stage 1 finds non-lexical over-correction below a few percent of test strings, there is no problem to solve.