Teaching Text Recognizers to Ignore Fonts They Have Never Seen

Scene text recognizers, the models that read words off signs, packages and dashcam frames, are almost always trained on millions of computer-rendered word images. They then fail badly on real photos. One documented failure is especially telling. A recognizer trained on five fonts cannot read a sixth font at all, which suggests it memorizes the outlines of particular glyphs rather than learning what an "a" is (Using Human Psychophysics to Evaluate Generalization in Scene Text Recognition Models). We propose to attack this font wall directly. During training, a small adversary tries to guess which font family each individual character came from, given which character it is. The recognizer is penalized whenever the adversary succeeds. Characters are located with the alignment that the CTC loss already computes, so no attention module or character boxes are needed. The model sees no target-domain data at any point. This is domain generalization, meaning training on several source domains so that the model works on unseen ones, rather than domain adaptation.

The opportunity is that existing text-specific methods all need target data. The general-purpose generalization methods, meanwhile, mostly perturb image style, and style is not where the font wall lives. A colleague's two-sentence version would be this: split your synthetic training data by font family and add a per-character, label-conditioned domain adversary so character features stop encoding font identity. Then test whether this reads held-out fonts and real benchmarks better than simply rendering more fonts or mixing image styles.

Core Claim

The claim has three parts. First, glyph shape is a geometric shift. A method that perturbs only the Fourier amplitude spectrum, the part of an image's frequency decomposition that carries color, contrast and texture, cannot simulate it, because a font change mostly alters the phase spectrum, which carries edges and layout. Second, a domain-adversarial loss can remove font identity from character features without collapsing different letters together. To do so it must be applied per character and conditioned on the character label, and its domains must be source font families rather than an unavailable target. Third, the CTC alignment, which assigns each column of the feature map either to a character or to "blank," provides the character localization this needs. It is more robust on curved text than the attention-based character extraction that prior adversarial text work relied on.

Concretely, let $h_t$ be the encoder feature at column $t$, let $c_t$ be the character that the best CTC alignment of the ground-truth string assigns to $t$ (blank columns are skipped), and let $d$ be the font-family domain of the training image. A discriminator $D$ receives $h_t$ together with an embedding of $c_t$ and predicts $d$. The encoder is trained through a gradient reversal layer, which passes features through unchanged on the forward pass and negates and scales the discriminator's gradient by $\lambda$ on the backward pass:

$$ \min_{\theta}\max_{D}\;\; \mathcal{L}_{\text{CTC}}(\theta) \;-\; \lambda \,\mathbb{E}_{t:\,c_t\neq\text{blank}}\big[-\log D(d \mid h_t, c_t)\big]. $$

Here $\theta$ are the recognizer's parameters. Because $D$ is told the character, the encoder only needs to make each letter's features indistinguishable across fonts. This targets the class-conditional distribution rather than the marginal one, and avoids the known failure where marginal alignment pulls distinct classes into the same region.

Relation to Prior Work

Four prior text-reading methods motivate the design, but all assume target data. GA-DAN translates synthetic images toward a target domain with separate geometry and appearance generators, reaching 43.1% on curved CUTE80 against 31.9% for CycleGAN. It shows that geometry matters, but it works in pixel space and must be retrained for every target. SSDAN aligns attention-extracted character features between source and unlabeled target using covariance matching, gated by prediction confidence. FASDA shows that adversarial alignment must happen per character, because the number of possible word strings grows exponentially with length. It needs 10–100 labeled target images, and its authors report that its attention-based character features break on curved text. StrDA uses a pair of domain discriminators to order 2 million unlabeled real images into an easy-to-hard curriculum for multi-round self-training.

We borrow two ideas from this group: alignment at the character level (from FASDA and SSDAN) and the discriminator machinery. We drop the target domain entirely. We also replace attention with CTC alignment, which the psychophysics study found more robust to string length and to non-letter clutter beside the word.

The domain-generalization literature supplies the comparison methods and also limits the design. FACT mixes Fourier amplitudes across domains and keeps phase fixed, while a teacher network supplies a consistency target. It was tested only on classification, and its authors note that amplitude can carry task-relevant contrast. FAN-Net combines amplitude normalization with a reversal-layer domain classifier for medical segmentation across hospital sites. It is the closest "adversarial plus generalization" recipe, but its domain classifier is unconditional and operates on whole feature maps. Episodic Training for Domain Generalization pairs feature extractors and classifiers trained on different source domains, gaining 3.3% over pooled training on the PACS image-classification benchmark. None of these has been run on a variable-length sequence decoder.

Research Program

We will build a rendered training corpus in which every image carries a font-family label. Families are grouped coarsely: serif, sans, monospace, script, display, stencil-like, and so on. Each family is rendered with backgrounds and degradations in the style of MJSynth and SynthText. This gives enough source domains for leave-family-out evaluation. The recognizer is a standard four-stage pipeline from What Is Wrong With Scene Text Recognition Model Comparisons?: a thin-plate-spline rectifier that flattens curved or tilted text, a ResNet feature extractor, a bidirectional LSTM, and a CTC prediction head (TPS-ResNet-BiLSTM-CTC). This keeps our results directly comparable with that paper's protocol.

The discriminator is a two-layer network on $[h_t; \text{emb}(c_t)]$. Alignments are recomputed every step from the current model, and the adversarial weight $\lambda$ is ramped up from zero, so early, poor alignments carry little weight. Analysis has two parts. A linear probe for font family on frozen character features should fall toward chance on held-out fonts. A probe for character identity should stay high, confirming that letters have not collapsed together. A secondary arm swaps CTC for attention-based character features, which tests whether CTC localization is load-bearing.

Baselines

The naive reference is empirical risk minimization: the same CTC recognizer trained on the same pooled corpus, with the same augmentation, step count and tuning budget. Any gain we report is measured against this.

The strongest published zero-target-data system on the standard real benchmarks is TPS-ResNet-BiLSTM-Attn. It reports 84.1% average word accuracy across the standard benchmarks in What Is Wrong With Scene Text Recognition Model Comparisons?. We will retrain it on our corpus under identical compute, and separately on MJ+ST to reproduce its reported setting. Beating it means a higher average on the same seven benchmarks with the cheaper CTC decoder. Even matching it would be notable, given the latency advantage of CTC over attention noted in the psychophysics study.

The strongest generalization competitor is FACT-style amplitude mixing with teacher consistency, ported to CTC. No reported result was found on Lacuna for amplitude mixing on any text-recognition benchmark, so we run it ourselves with an equal tuning budget over the mixing ratio and the cutoff radius. We also run FAN-Net's unconditional reversal-layer domain classifier on pooled feature maps; no reported result was found on Lacuna for this on text recognition.

Three ablations isolate the key ingredient. The first removes the character conditioning, giving a marginal discriminator. The second keeps everything but shuffles the font labels, which tests whether the adversary helps only as a generic regularizer. The third adds amplitude mixing on top of our method, which tests whether the photometric and font effects add up.

StrDA and SSDAN are reported as target-data upper references rather than competitors. SSDAN lifts IIIT5K from 81.1% to 83.8% using unlabeled target images. StrDA reports gains of 7.2 and 7.6 points on Union14M but no absolute accuracy that we found.

Why not just…?

Why not just render more fonts? This is the real threat. Data diversity may be all that matters, and adversarial training may amount to an expensive substitute for it. We settle this with a scaling curve. We train empirical risk minimization with $K$, $2K$ and $4K$ source families and our method with $K$, then measure held-out-family accuracy. If empirical risk minimization at $2K$ matches our method at $K$, the contribution reduces to a data-efficiency trick at best. If it matches with no fonts held out, we abandon the claim. The psychophysics result that five fonts give no transfer to a sixth shows that small font counts fail. It does not tell us where pooled data saturates.

Why not just use Fourier amplitude mixing? Amplitude mixing changes contrast and texture while holding phase fixed, and phase is where glyph shape lives. The FACT authors themselves warn that amplitude can carry task-relevant contrast, which in text means the contrast between characters and their background. We therefore predict that mixing helps on blur- and lighting-dominated sets such as SVT and RoadText-1K, and does nothing on held-out fonts. The combined run tests whether the two methods address separable shifts.

Why not just use episodic mismatched-partner training? Episodic training needs a separate decoder per domain and was shown only for classification, where heads are cheap. With a CTC head it is feasible, so we include Epi-FCR-style training with one CTC head per font family as an additional baseline under the same budget. If it matches our method, the lesson is "simulate domain shift" rather than "adversarial removal of font identity," and we will report it that way.

Why not just fine-tune or self-train on the target? These methods answer a different question. FASDA and StrDA show that target data helps, but deployment domains such as new signage, packaging or industrial codes often arrive unannounced. Fine-tuning also collapses at 100 labeled samples, falling to 1.8% on batch numbers in Text Recognition in Real Scenarios with a Few Labeled Samples. We additionally test whether our model is a better starting point for StrDA, which would make the two complementary.

Proposed Experiments

The first, controlled experiment is leave-one-family-out word accuracy on our rendered corpus, following the psychophysics protocol of Using Human Psychophysics to Evaluate Generalization in Scene Text Recognition Models, including its string-length and clutter-character probes. Success means a large held-out-font gain over empirical risk minimization while in-font accuracy stays flat.

The real-world transfer experiment trains on synthetic data only. We evaluate on IIIT5K, SVT, IC13 and IC15, using the standardized filtering from the benchmark-comparison paper so the 158-image IC13 discrepancy does not contaminate results. We also evaluate on CUTE80 against GA-DAN's 43.1%, keeping in mind that GA-DAN used target data. Union14M covers artistic text, where font shift should dominate. RoadText-1K, whose pretrained-model accuracy is reported as low as 29% in RoadText-1K, covers motion blur and glare, where we expect amplitude mixing rather than our loss to help.

The decisive checks are three. Held-out-font gains must exceed what doubling the number of fonts buys. The label-shuffle and unconditional-discriminator ablations must lose most of the gain. Finally, the improvement on Union14M must exceed the improvement on RoadText-1K, as the font-versus-photometric account predicts. If the gains are uniform across benchmarks, the adversary is acting as a generic regularizer, and the mechanistic claim fails.