Reading Font Style Out of Frozen Vision Models by Subtracting the Character

Large pretrained vision models are bad at telling fonts apart. When shown the word "Arial" set in Times New Roman, they report what the text says rather than how it is drawn. This failure has been measured directly (Texture or Semantics?), and it is the main reason few-shot font generators still train their own style encoders from scratch on hundreds of paired fonts. Few-shot font generation means producing thousands of characters in a new typeface from roughly 8 to 16 examples. We propose that the failure comes from how the style signal is read out, not from what the model represents. Stroke weight, terminals, and slant are likely still present in the intermediate patch features of a frozen backbone. They are buried under a much larger signal for which character is shown. In few-shot font adaptation we always know which characters the reference glyphs are, so we can render those same characters in a neutral font, pass both through the frozen model, and keep only the difference. The character identity cancels and what remains is style.

The core claim is that a content-subtracted residual, computed layer by layer on the patch features of a frozen self-supervised or contrastive vision transformer, gives a style representation for logographic scripts that needs no backbone training. This representation should (a) beat raw frozen features by a wide margin on style retrieval and on the Stroop-style recognition test, and (b) act as a drop-in style encoder for a few-shot Chinese font generator. In that second role, it should match encoders trained from scratch when many training fonts are available, and beat them when training fonts are scarce, when the test script differs from the training script, or when the target style is an extreme one such as calligraphy. If the residual from a randomly initialized backbone, or from plain pixels, works just as well, the claim is false. The design includes those controls.

What exists and what this changes

Three lines of work sit closest. The first trains a dedicated style extractor and then freezes it. Font Representation Learning via Paired-glyph Matching trains a ResNet-18 contrastively so that different letters in the same font count as matches. It reaches 91.82% mean retrieval accuracy on the Open Font Library and then serves as a fixed style encoder for transfer. FontAdapter trains a resampler module on synthetic text-only images and freezes it before training on scene text. Both freeze an extractor they trained themselves for typography. We freeze a general-purpose backbone that never saw a font-specific objective. PGM was also tested only on Latin letters with a single global vector, and we target CJK scripts with local, patch-level residuals.

The second line adapts the foundation model itself. FontCLIP fine-tunes CLIP on font attribute prompts and raises attribute correlation from 0.159 (raw CLIP) to 0.723. That result motivates our work, because it shows raw pooled CLIP embeddings carry little typographic signal. It also constrains our work: if fine-tuning is needed to get that signal, our claim fails. The difference is that FontCLIP's baseline is the final pooled embedding. We read intermediate patch tokens and remove content explicitly, and nobody in the file has measured that.

The third line documents the failure we are explaining. Texture or Semantics? finds that 13 vision-language models reach about 30% on the easy version of its Font Recognition Benchmark and about 15% on the hard Stroop version, with under 4% gain from few-shot prompting. Its attention maps show the models looking at semantic tokens rather than stroke edges. We read this as evidence about the language readout. Our first experiment tests whether the frozen visual features themselves are equally blind. If they are, the paper becomes a clean negative result.

On the generation side, the field's own diagnosis supplies the method. IF-Font and CF-Font both identify source leakage: the neutral source font's shapes bleed into the output. Our subtraction is the encoder-side mirror of that problem. It removes the reference character's content from the style code, which should reduce how much reference-character structure the style code carries into new characters. Few-Shot Font Generation by Learning Fine-Grained Local Styles (FSFont) shows that local cross-attention over reference patches beats global style vectors. We keep that kind of local aggregation and change only what the patches contain.

Research Program

The representation works as follows. Let $\phi_\ell(x)$ be the patch tokens at layer $\ell$ of a frozen backbone for glyph image $x$. For a reference glyph $r$ of character $c$, render $c$ in a fixed neutral font to get $n_c$. Same-character glyphs in different fonts are not pixel-aligned, so each reference patch is matched to neutral patches through a fixed, parameter-free soft alignment (attention computed on frozen features) before subtracting. The style residual is

$$s_\ell(r) = \phi_\ell(r) - A_\ell(r, n_c)\,\phi_\ell(n_c),$$

where $A_\ell$ is the row-softmax of scaled dot products between the two sets of patch tokens. The only trained parts are a small per-layer projection and a layer-mixing weight on $s_\ell$. These are fit with the paired-glyph contrastive objective from PGM, in which different characters in the same font are positives, and the backbone never receives gradients.

The work proceeds in two stages. The diagnostic stage asks where style lives. On frozen CLIP and a self-supervised ViT, we measure same-font retrieval for every layer under three readouts: pooled raw features, raw patch features, and residual patch features. The task is to match a query character in font $f$ against a gallery of different characters across fonts. We run this on a Chinese font collection and on the Latin Open Font Library, where PGM's 91.82% is the reference point. The generation stage replaces the style encoder of one fixed few-shot generator with the residual reader, leaving the generator, losses, and training data unchanged. We use a CF-Font-style content-fusion generator with FSFont-style cross-attention aggregation. Holding everything else fixed means any change in outcome is attributable to the style representation.

Baselines

The naive reference point is a linear probe on the frozen pooled embedding, which corresponds to the raw-CLIP numbers in FontCLIP (0.159 attribute correlation, 67.68% / 54.92% in- and out-of-domain pairwise accuracy). It is run with the same contrastive head and the same data as our method. Beating it is necessary but not impressive. The next reference is raw frozen patch features with an identical projection head and no subtraction. This is the ablation that isolates the key ingredient: if subtraction does not add substantially on top of patch features alone, the content-cancellation idea is not doing the work.

The strongest generation competitor is a style encoder trained from scratch inside the same generator, which is how FSFont, CF-Font and VQ-Font all work. The strongest reported number in the file is from VQ-FONT: UFUC L1 of 0.0621 and LPIPS of 0.103 on its 382-font benchmark. UFUC means unseen fonts and unseen characters. We will retrain VQ-Font with its public protocol rather than copy its numbers. Our reader and the from-scratch encoder get identical generator, data, training steps, and hyperparameter search budget. For content and style fidelity we also report the harmonic mean accuracy of LF-Font, which reports 83.2% against DG-Font's 63.1%.

The strongest backbone-adaptation competitor is LoRA tuning of the same backbone on the same contrastive objective, given the same data and step budget. This is the FontCLIP-style route made directly comparable. For instant adaptation we compare against a FontAdapter-style trained-then-frozen resampler. Its paper reports about 11 seconds per font against 40 minutes for per-font LoRA. Its Max-IoU and HOG-similarity numbers are not in the file, so we rerun it.

Beating the from-scratch encoder at the full 300+ training fonts is not required. The method counts as winning on two conditions. At full data, it must stay within noise of the from-scratch encoder on UFUC L1 and LPIPS. In at least two of the three stress regimes (20–50 training fonts, cross-script transfer, calligraphic test fonts), it must clearly beat that encoder.

Why not just…?

Why not just train the style encoder from scratch, as every font generator already does? With roughly 400 paired training fonts this may well be equal or better, and we treat that as a real threat. The case for a frozen backbone rests on regimes the file flags as open. One is paired-data dependency (LF-Font). Another is extreme and calligraphic styles, where FSFont, CF-Font and LF-Font all report degradation. The training-font sweep and cross-script test settle this directly. If the from-scratch encoder wins everywhere, we report it.

Why not just fine-tune the backbone with LoRA, as FontCLIP fine-tunes CLIP? That is the matched baseline above. Fine-tuning on a few hundred Roman fonts left FontCLIP with attribute entanglement and an out-of-domain correlation of only 0.404. This suggests that adapting the backbone on narrow font data trades away generality, which matters for cross-script transfer. If LoRA beats subtraction on cross-script retrieval at equal budget, the "frozen" part of our claim fails.

Why not just subtract pixels, or use any network? The cancellation trick could work for reasons unrelated to pretraining, because a glyph minus its neutral rendering is already a crude style image. We therefore run the identical residual on raw pixels (with the same alignment) and on a randomly initialized backbone of the same architecture. If either matches the pretrained residual, the result is about subtraction, not about frozen features, and we will say so.

Why not just prompt a vision-language model with examples? Texture or Semantics? already shows few-shot prompting gains under 4% and roughly 15% accuracy on the hard benchmark. We include its hard version as a direct test of our representation, not as an option to pursue.

Decisive Experiments

The first check comes before any generation work. On the 225-image hard version of the Font Recognition Benchmark, a nearest-neighbor classifier on residual features must clearly exceed the roughly 15% reported for the best vision-language models. It must also exceed raw patch features from the same backbone. The benchmark's text content is known, so the neutral renderings are exact. If the residual does not separate from raw features here, the project stops at a diagnostic paper on where typographic information disappears in frozen models.

The second check is the layer-resolved retrieval curve on Chinese fonts. We expect residual retrieval to peak in early-to-middle layers, where pooled features are weakest. This would show directly that the signal is present but not read out.

The third check is generation on the CF-Font 300-font benchmark (16 references, 6,446 characters) and the VQ-Font UFUC split, under the matched protocol. We sweep training-font count and report L1, LPIPS, FID, and harmonic-mean accuracy computed with a held-out character recognizer and a held-out font classifier. This follows the warning in Rethinking HTG Evaluation that FID and SSIM can reward blurry approximations. Cross-script transfer trains on Chinese and tests on the 201-font Korean set used by Diff-Font and on Latin letters. A separate calligraphy-heavy test subset probes extreme styles.

The method also needs no font-specific backbone, so adapting to a new font requires only frozen forward passes and a cross-attention readout. We will report per-font adaptation time alongside FontAdapter's 11 seconds.