Abstract
Traditional handwritten text recognition (HTR) systems primarily rely on visual features, often overlooking the underlying geometric structures of handwriting. This paper presents the Graph-Enhanced Cross-Modal Mutual Learning (GCM) framework, which integrates visual imagery with geometric "skeletons." By employing two parallel decoding branches that engage in mutual distillation via Kullback-Leibler (KL) divergence, GCM achieves state-of-the-art performance on multiple benchmarks. Notably, the framework maintains inference efficiency by activating only a single branch during production.
Introduction
Most modern HTR systems treat handwriting as a purely visual task, utilizing Convolutional Neural Networks (CNNs) or Transformers to model relationships between adjacent pixels. However, human perception of handwriting is inherently structure-aware; we recognize the strokes and geometric formations that constitute characters.
While geometric graphs (skeletons) offer an intuitive representation of these structures, integrating them with visual data is challenging. Previous multi-modal approaches often suffer from "modality competition," where one modality dominates training, or "attention drift," where the model loses spatial focus. The GCM framework addresses these issues by creating a structure-aware system that effectively fuses visual and geometric information through mutual learning.
Methodology
The GCM framework represents a significant architectural evolution in HTR. As illustrated in Figure 1, while traditional models rely on single-modality attention or simple fusion, GCM utilizes a cross-modal approach where geometric and visual contexts guide one another.
The system is composed of three primary components: Independent Multi-Modal Encoding, Cross-Modal Decoders with Cascaded Attention, and Bidirectional Mutual Learning.
1. Independent Multi-Modal Encoding
The system first converts handwriting images into skeleton-based graphs. It then employs two specialized encoders: a Convolutional Recurrent Network (CRN) for visual features and a Pyramid Graph Transformer (PyGT) for geometric features. This ensures that both the pixel-level appearance and the stroke-level structure are captured independently.
2. Cross-Modal Decoders with Cascaded Attention
Instead of a simple feature fusion, GCM introduces two parallel decoding branches:
- G → V Branch: Geometric context is used to guide the visual attention mechanism.
- V → G Branch: Visual context guides the geometric attention mechanism. This cascaded attention significantly reduces attention drift by using the geometric graph as a "map" for visual focus.
3. Bidirectional Mutual Learning
During the training phase, the two branches are forced to reach a consensus through a Kullback-Leibler (KL) divergence loss. This acts as a mutual distillation mechanism where each branch "teaches" the other at every time-step. To ensure stability, uni-modal loss objectives are included to prevent the model from ignoring one modality. Crucially, while both branches are used for training, only the inference branch is activated during deployment, providing a performance boost without additional computational overhead.
Experimental Results
The GCM framework was evaluated on four diverse benchmarks, consistently outperforming previous state-of-the-art (SOTA) methods without the use of extra synthetic data:
- IAM (English): Achieved a Character Error Rate (CER) of 5.8% and a Word Error Rate (WER) of 14.9%, a notable improvement over visual-only baselines (6.4%–7.9% CER).
- RIMES (French): Achieved a CER of 2.4%.
- IAHEW-UCAS2016 (Online HTR): Reached a CER of 1.04%, demonstrating flexibility for both online (stroke-based) and offline (image-based) data.
- ICDAR2013 (Chinese): Hit 97.12% accuracy, outperforming specialized radical-based recognition models by effectively capturing complex character structures.
Discussion and Conclusion
The GCM framework successfully bridges the gap between visual and structural representations in HTR. By using geometric graphs to guide visual attention and employing mutual distillation, the model captures complementary patterns that single-modality systems miss.
One current limitation is the reliance on clean backgrounds for accurate skeleton extraction. In complex scene text recognition (e.g., handwritten signs in cluttered environments), noise and occlusions make graph extraction difficult. Future research will focus on extending this structure-aware approach to unconstrained scene text environments.
Summary updated