Abstract
Handwritten Mathematical Expression Recognition (HMER) is a complex task due to the non-linear structure of mathematical notation and the inherent visual ambiguity of handwritten symbols. Traditional encoder-decoder models often struggle to distinguish between visually similar characters without explicit contextual knowledge. This paper introduces the Semantic Aware Module (SAM), a novel approach that incorporates "linguistic" relationships between mathematical symbols into the learning process. By constructing a semantic graph based on symbol co-occurrence statistics, the proposed method supervises feature learning to improve recognition accuracy. Experimental results on the CROHME and HME100K benchmarks demonstrate that this approach achieves new state-of-the-art performance while maintaining zero additional computational overhead during inference.
Introduction
Recognizing handwritten mathematical expressions is significantly more challenging than standard optical character recognition (OCR). Mathematical notation is two-dimensional, and many symbols are visually nearly identical—for instance, a handwritten "b" can be indistinguishable from a "6," or a capital "B" from the Greek letter "$\beta$."
While modern deep learning models, particularly those utilizing encoder-decoder architectures with attention mechanisms, have made strides in mapping images to LaTeX code, they often treat symbols as isolated visual entities. These models frequently lack an explicit understanding of semantic dependencies—the statistical likelihood that certain symbols appear together in a mathematical context. This paper addresses this gap by proposing Semantic Interaction Learning (SIL), which teaches the model to utilize the mathematical "vocabulary" to resolve visual ambiguities.
Methodology
The core of the proposed method is the Semantic Aware Module (SAM), a "plug-and-play" component designed to enhance existing HMER backbones (such as CAN or DWAP). The methodology consists of three primary stages:
1. Construction of the Semantic Graph
To capture the relationships between symbols, the researchers analyzed training datasets to calculate the conditional probability of symbol co-occurrence. This data is used to build a correlation matrix, which serves as a semantic graph where nodes represent symbols and edges represent the strength of their relationship. This graph acts as a repository of mathematical "collocations."
2. The Semantic Aware Module (SAM) Architecture
The SAM is integrated into the training pipeline and consists of two parallel branches:
- Visual Branch: This branch projects visual feature vectors from the encoder into a high-dimensional semantic space.
- Classification Branch: This branch projects classification features into the same semantic space.
3. Joint Optimization and Feature Shaping
During training, the model is optimized using a multi-part loss function. In addition to the standard Cross-Entropy loss for symbol prediction, the model is tasked with minimizing the cosine distance between its projected vectors to align them with the pre-calculated Semantic Graph. This forces the visual encoder to produce features that are not only visually descriptive but also semantically meaningful.
Crucially, the SAM is only utilized during the training phase. Once the encoder has been "taught" these semantic relationships, the SAM is discarded, ensuring that the model achieves higher accuracy during inference without any increase in latency or parameter count.
Experimental Evaluation
Datasets
The researchers evaluated the SAM-enhanced models on two distinct types of benchmarks:
- CROHME (2014, 2016, 2019): These are the standard benchmarks for HMER, consisting of relatively clean images synthesized from digital pen strokes.
- HME100K: A large-scale, real-world dataset containing 100,000 images. These photos feature varied lighting, motion blur, and complex backgrounds, providing a rigorous test of the model's robustness.
Key Results
The integration of the SAM module consistently improved the performance of baseline models across all metrics:
- State-of-the-Art Performance: When paired with the Counting-Aware Network (CAN) backbone, the method achieved an ExpRate (Expression Recognition Rate) of 58.0% on both CROHME 2014 and CROHME 2019, setting new benchmarks for the field.
- Robustness in Real-World Scenarios: On the challenging HME100K dataset, the SAM-enhanced model showed significant gains, particularly on the "Hard" subset, where accuracy improved from 45.4% to 54.0%.
- Ambiguity Resolution: Qualitative analysis confirmed that the module successfully uses context to tip the scales in favor of mathematically logical symbols. For example, in cases where a symbol was visually ambiguous between a "b" and a "6," the SAM module increased the model's confidence in the correct "b" prediction from 10.2% to 81.0% by analyzing the surrounding semantic context.
Discussion and Conclusion
This research demonstrates that mathematical expressions possess a "linguistic" structure that can be leveraged to improve machine recognition. By treating math as a language with specific symbol dependencies, the Semantic Aware Module provides a powerful supervisory signal that shapes the encoder's feature representation.
Limitations and Future Directions
Despite its success, the current approach relies on static statistics; the semantic graph is built from global training data and does not adapt to specific mathematical sub-fields (e.g., the symbol distributions in linear algebra versus calculus). Additionally, while the SAM excels at symbol identification, further work is needed to address the structural parsing of highly complex 2D notations, such as deeply nested fractions or large matrices. Future research may explore dynamic semantic graphs or extending this methodology to other multi-dimensional OCR tasks where contextual relationships are vital for resolving visual uncertainty.
Summary updated