SSAN: A Symbol Spatial-Aware Network for Handwritten Mathematical Expression Recognition

Haoran Zhang, Xiangdong Su, Xingxiang Zhou, Guanglai GaoAAAI 2025 · DOI

Introduction

Handwritten Mathematical Expression Recognition (HMER) presents a unique challenge compared to standard optical character recognition (OCR). While text is typically a one-dimensional sequence, mathematical formulas are inherently two-dimensional. The spatial arrangement of symbols—such as superscripts, subscripts, and fraction bars—is critical to the formula's semantic meaning.

Current state-of-the-art HMER models often utilize an encoder-decoder architecture with attention mechanisms. However, these models frequently suffer from "attention drift," where the decoder loses track of the precise spatial context of symbols, leading to structural errors. To address this, the Symbol Spatial-Aware Network (SSAN) introduces an auxiliary task that explicitly supervises the model's understanding of symbol spatial distribution.

Methodology

The core innovation of SSAN is the integration of a spatial-aware auxiliary task within a multi-task learning framework.

Symbol Spatial Distribution Mapping

The model is trained to predict a "Symbol Spatial Distribution Map" alongside the standard LaTeX sequence. To generate ground-truth maps, the researchers utilize printed templates of the LaTeX labels. Each symbol in the printed image is identified and replaced with a 2D Gaussian distribution map centered on its location. This creates a "heat map" representing the ideal spatial arrangement of symbols.

Comparison between formula images and their corresponding symbol spatial distribution maps.

Coarse-to-Fine Training Strategy

Recognizing that human handwriting rarely aligns perfectly with printed templates, the authors implemented a two-stage training strategy. In the "Coarse" stage, the model learns to predict the spatial map of the printed template. In the "Fine" stage, the model refines these predictions using attention maps generated by its own decoder, ensuring that spatial awareness is self-consistent with the actual handwriting input.

The two-stage training pipeline of SSAN, illustrating the coarse-to-fine strategy and the integration of the spatial-aware network.

SSAN Architecture

The SSAN consists of Symbol Spatial-Aware (SSA) submodules that incorporate a Top-Down Attention Module (TDAM). This module fuses high-level semantic information with low-level local features, allowing the network to focus on critical regions within the formula image. Additionally, an attention-guided symbol masking strategy is employed to force the model to learn more discriminative features, helping it distinguish between visually similar symbols.

Experiments and Results

The SSAN was evaluated on the CROHME 2014, 2016, and 2019 competition datasets, as well as the large-scale HME100K dataset.

Performance Gains

The addition of the SSAN module provided significant improvements over baseline models. On the CROHME 2014 dataset, the DWAP+SSAN model achieved a recognition accuracy of 60.85%, representing a nearly 10% absolute improvement over the baseline DWAP (51.48%). The model also demonstrated superior performance compared to other recent state-of-the-art methods like BPD-Coverage and CAN-ABM.

Structural Robustness and Efficiency

Deep-dive analysis revealed a 13% increase in accuracy specifically for formulas with complex structural elements like nested superscripts and subscripts. Despite the increased complexity, the model remains efficient; inference speed only decreased slightly from 20.52 to 19.18 FPS, maintaining practicality for real-world applications.

Conclusion

By treating spatial awareness as an explicit supervised task, SSAN effectively mitigates the problem of attention drift in HMER. The use of printed templates to bootstrap spatial understanding, combined with a coarse-to-fine refinement strategy, allows the model to handle the inherent variability of human handwriting while maintaining high structural accuracy. Future work may involve tighter integration of these spatial maps directly into the decoding process to further enhance real-time recognition.

Summary updated

BibTeX
@inproceedings{zhang2025ssan,
  author = {Haoran Zhang and Xiangdong Su and Xingxiang Zhou and Guanglai Gao},
  title = {{SSAN: A Symbol Spatial-Aware Network for Handwritten Mathematical Expression Recognition}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.1609/aaai.v39i21.34396},
}
Figure