Logographic characters such as Chinese Hanzi or ancient Oracle Bone script are built from smaller parts, called radicals, placed in a two-dimensional layout: one radical to the left of another, one stacked on top of another, one enclosing another. A character can therefore be written as a small tree whose leaves are radicals and whose internal nodes say how their children are arranged. Recent systems learn to match character images to these trees. They do this so that a character never seen in training can still be recognized from its dictionary decomposition. The current best of these matches only one summary vector of the whole image against one summary vector of the whole tree. That throws away the most useful fact the tree carries, which is where in the image each part should be. We propose to use the layout operators in the tree to place a soft spatial window for every tree node, and to align each node's embedding with the image features pooled inside its window. The bet is that this matters most when real images are scarce. A radical that appears in hundreds of characters then gets hundreds of grounded training examples, even if every character containing it has only one or two images.
Put in two sentences: the tree already tells you where each radical sits, so use it to tell the image encoder where to look, and supervise every radical and sub-layout rather than only the whole character. This should narrow the gap between image embeddings and tree embeddings and improve recognition of rare and unseen characters in long-tailed ancient-script collections.
Core Claim
Global image-to-tree contrastive alignment leaves the fine structure of the tree ungrounded. Node-level alignment, with spatial windows derived from the tree's own layout operators, should produce three effects. It should give higher zero-shot and few-shot accuracy on long-tailed ancient-script benchmarks. It should give a measurably smaller gap between the image and tree embedding distributions. And the advantage should grow as the number of training images per class shrinks. The layout-derived window is the key ingredient. If uniform windows, or windows learned freely without the layout prior, do as well, the claim is false.
Where This Sits
The closest work is FT-CLIP from Improving Chinese character representation with formation tree. It encodes characters as formation trees with a tree Transformer that uses subtree-restricted attention and learnable "azimuth" position embeddings. It aligns them to a Vision Transformer image encoder with CLIP-style contrastive learning, which pulls matched image–tree pairs together and pushes mismatched pairs apart. It reaches up to 78.96% zero-shot accuracy on ICDAR2013 with 2,755 training categories. Its own ablations show that removing the tree structure hurts, which is why we keep the tree rather than flattening it. FT-CLIP aligns only the global representations, however. It is reported as "not leading" on scene characters in the CTW benchmark, and it is not evaluated with a few images per class. We keep its encoders and add grounding at the level of individual nodes.
HierCode, from HierCode: A Lightweight Hierarchical Codebook for Zero-shot Chinese Text Recognition, replaces an 87,000-way classifier with multi-hot codes built from layout bits and radical codes. It reports up to 5.91% zero-shot gains and notes that the approach collapses for single-radical characters. That collapse also bounds our method, and we report on it separately. Like FT-CLIP, HierCode matches whole-character codes and not localized regions.
Two works pair radical decomposition with long-tailed, low-sample ancient scripts, and they are our strongest competitors. RZCR, from RZCR: Zero-shot Character Recognition via Radical-based Reasoning, extracts radical candidates and structural relations with attention blocks and then reasons over a hand-built character knowledge graph. It reaches 61.36% Top-1 on OracleRC, against 28.50% for ResNet and 36.79% for HDE-Net, and about 54% on unseen classes. The ACCID dataset paper, Toward Zero-shot Character Recognition: A Gold Standard Dataset with Radical-level Annotations, adds radical boxes and structure labels to 15,085 rubbing images. It splices annotated radicals into 46,800 synthetic samples and reaches 53.74% zero-shot accuracy on 800 unseen classes with a radical extractor plus dictionary reasoner. Both works decide on discrete radicals first and match strings afterwards. Both note brittleness when strokes merge across radical boundaries. Neither learns a shared embedding space for images and trees.
Our approach sits between these two families. It is a learned embedding space like FT-CLIP's, with supervision at the radical level like RZCR's and ACCID's. It needs no box annotations at training or test time.
Two further lines motivate the design without supplying it. Learning Robust Visual-Semantic Embeddings argues that pairwise contrastive losses pull matched points together without aligning the two distributions, which hurts on novel categories. A Visual Attention Grounding Neural Model for Multimodal Machine Translation finds that a single global visual vector tends to be ignored unless it is tied to regions. Separately, small-data work outside this domain finds that hard-coded structure substitutes for data. MiME (MiME) reports a 15% relative PR-AUC gain on the smallest subsets from a fixed clinical hierarchy. Unveiling COVID-19 from Chest X-ray with deep learning finds that spatial masking reduces shortcut learning and that ResNet-50 overfits small cohorts where ResNet-18 does not. These are analogies rather than evidence about characters. They are the reason we test the failure mode directly, by checking whether the model attends to the correct region, rather than assuming it.
Research Program
We start from a reimplemented FT-CLIP. It has a ViT image encoder producing patch features $p_j$ and a formation-tree Transformer producing an embedding $t_n$ for every node $n$, where leaves are radicals and internal nodes are layout operators.
We add a layout window module. The root's window $m_{\text{root}}$ covers the whole image. Each internal node splits its parent's window among its children according to its layout type: left–right, top–bottom, enclosure, or one of the other types in the 10-type inventory HierCode uses. The split uses a learnable ratio and edge softness per layout type. A small image-conditioned head predicts a bounded deformation of each split, so that a compressed left radical or an overhanging stroke can be accommodated. The deformation is penalized for drifting far from the layout prior. The region feature for node $n$ is $r_n=\sum_j m_n(j)\,p_j / \sum_j m_n(j)$, a window-weighted average of patch features.
Training minimizes
$$ \mathcal{L} = \mathcal{L}_{\text{global}}(z_{\text{img}}, t_{\text{root}}) + \lambda \sum_{n} \mathcal{L}_{\text{node}}(r_n, t_n), $$
where $\mathcal{L}_{\text{global}}$ is FT-CLIP's image–tree contrastive loss on the image summary $z_{\text{img}}$ and the tree root $t_{\text{root}}$. $\mathcal{L}_{\text{node}}$ is a contrastive loss in which each region feature must pick out its own node embedding among node embeddings drawn from the batch. Its negatives include the same radical type at other positions and other radicals at the same position, so it is forced to encode both identity and placement. $\lambda$ weights the node term.
For small-sample transfer, recognition uses nearest neighbors in the joint space. A class with zero images is represented by the embedding of its dictionary tree. A class with $k$ images is represented by that tree embedding averaged with the class's image prototypes.
We analyze three things beyond accuracy. The first is the image–tree modality gap, measured as the distance between modality centroids and as the accuracy of a linear probe trained to tell image embeddings from tree embeddings. The second is grounding fidelity: on ACCID, whose radical boxes are never used in training, we measure the overlap between learned windows and annotated boxes. The third is accuracy split by tree depth, so the single-radical degenerate case is reported on its own.
Baselines
The naive reference point is a ResNet-18 classifier with a prototype nearest-neighbor head, trained on the same capped data. It cannot recognize zero-shot classes and serves as the floor for the few-shot ones. We choose ResNet-18 because Tartaglione et al. find it more robust than deeper networks on small data. For reference, ResNet reaches 28.50% Top-1 on OracleRC as reported in RZCR.
The strongest learned-embedding competitor is FT-CLIP as described in Improving Chinese character representation with formation tree. It is the method our proposal extends and the only one that already aligns images to formation trees. We reimplement it with identical encoders, data, augmentation, step count and hyperparameter search budget, so that the node loss and window module are the only difference. Its reported 78.96% zero-shot accuracy on ICDAR2013 with 2,755 training categories is the anchor for checking our reimplementation. On that protocol, we count it as beaten if we match or exceed it. On the low-sample ancient-script splits, we count it as beaten if we show a gain larger than the seed-to-seed variation, and a gain that grows as images per class decrease.
The strongest system on the target data is the radical-reasoning family, RZCR from RZCR and the ACCID extractor-plus-dictionary baseline from Toward Zero-shot Character Recognition. Their reported numbers are 61.36% Top-1 and about 54% unseen-class accuracy on OracleRC for RZCR, and 53.74% zero-shot on ACCID's 800 unseen classes and 60.28% on full ACCID for the ACCID baseline. On those exact protocols, beating them means exceeding these figures. On our new capped-shot splits, no reported result was found on Lacuna, so we will rerun public implementations if available under the same splits. Otherwise we report against their published protocol only and say so.
The decisive ablations keep everything fixed and change only the window. In the first, the node loss is kept but every window is uniform over the whole image. The second uses free attention windows with no layout prior. The third uses the layout prior without the image-conditioned deformation. The fourth is an oracle that uses ACCID's annotated boxes as windows, to bound how much better grounding could help.
Why not just…?
Why not just add ACCID-style splicing augmentation to FT-CLIP? This is a real threat. Splicing creates many new image–tree pairs, and more pairs might close the gap with no architectural change. Toward Zero-shot Character Recognition itself notes that box splicing ignores how radicals deform in context, so its synthetic images may teach the wrong shapes, but that is a hypothesis rather than a result. We will run FT-CLIP with splicing, our method without splicing, and our method with splicing. If FT-CLIP with splicing matches our method without splicing on unseen ACCID and OracleRC classes, node grounding is not earning its keep.
Why not just train a radical detector and match against the dictionary, as RZCR does? Detect-then-match makes hard decisions early. Both RZCR and the ACCID paper report that it degrades when strokes merge across radical boundaries or rubbings are corroded. It also depends on a hand-built knowledge graph. Our method keeps soft windows and a continuous embedding, so a partially merged radical can still match approximately. The test is a held-out subset of ACCID characters with heavy corrosion or merged components, identified via annotation overlap, compared head-to-head. If the radical-reasoning systems still win there, our claimed robustness advantage is wrong.
Why not just fine-tune a large pretrained vision model with prototypes for the few-shot classes? This handles the few-shot case well and is cheap. It cannot represent a class with zero images, which is most of the long tail in OracleRC. Tartaglione et al. also report that deeper models overfit small cohorts. We include a fine-tuned pretrained ViT with prototypes at 1, 3 and 5 shots. If it matches our method on few-shot classes, we will state that our contribution is confined to zero-shot and very-low-shot classes.
Why not just flatten the tree into a radical sequence? FT-CLIP's ablations already show that removing subtree and azimuth encoding lowers its 78.96% peak. Radical Embedding and Sub-character Neural Language Modelling in Japanese argue that flattening discards meaning-bearing layout. Our method also needs the layout operators to place windows at all, so a flat-sequence variant is run only as a sanity check.
Proposed Experiments
The primary benchmarks are ACCID and OracleRC, the two long-tailed ancient-script sets in the file. ICDAR2013 with CASIA-HWDB1.1 training serves as a check against FT-CLIP's published protocol. CTW serves as a stress test for scene text, where FT-CLIP is reported as not leading; no number is reported in the file, so we report our reproduction.
On ACCID, we keep the published 800-class unseen split and cap the seen classes at 1, 3 and 5 images per class as well as full data. This produces a curve of zero-shot and few-shot Top-1 accuracy against images per class, averaged over at least three seeds. On OracleRC, whose many single-sample classes already form a natural low-sample regime, we use RZCR's protocol and add a few-shot split for the same caps.
The metrics are Top-1 accuracy on seen, few-shot and unseen classes, reported separately for single-radical characters. Alongside these we report the modality-gap measures and window-to-box overlap on ACCID.
The checks that would settle the question are as follows. First, the gain of our method over reimplemented FT-CLIP should widen as images per class fall from full data to one. Second, uniform-window and free-window ablations should recover little of that gain. Third, the reduction in the linear-probe modality gap should track the accuracy gain across ablations; if the gap shrinks without accuracy gains, the gap is not the bottleneck. Fourth, learned windows should overlap annotated radical boxes well above the overlap expected by chance. Finally, FT-CLIP with splicing should not close the gap. If the first or second check fails, we conclude that global alignment was not the limiting factor and that the layout prior adds nothing beyond more supervision. That would be a clean negative result.
References
- Improving Chinese character representation with formation tree
- HierCode: A lightweight hierarchical codebook for zero-shot Chinese text recognition
- RZCR: Zero-shot Character Recognition via Radical-based Reasoning
- Toward Zero-shot Character Recognition: A Gold Standard Dataset with Radical-level Annotations
- Learning Robust Visual-Semantic Embeddings
- A Visual Attention Grounding Neural Model for Multimodal Machine Translation
- MiME: Multilevel Medical Embedding of Electronic Health Records for Predictive Healthcare
- Unveiling COVID-19 from CHEST X-Ray with Deep Learning: A Hurdles Race with Small Data
- Radical Embedding: Delving Deeper to Chinese Radicals
- Sub-character Neural Language Modelling in Japanese