Introduction
Handwritten Mathematical Expression Recognition (HMER) presents a unique challenge compared to standard optical character recognition (OCR). While text is typically linear, mathematical expressions are inherently two-dimensional, featuring complex spatial relationships such as superscripts, subscripts, fractions, and nested radicals.
Current attention-based models often struggle with two specific issues: the "Scale Problem," where fixed-height normalization obscures small symbols like exponents, and "Attention Imprecision," where slight misalignments in the model's focus lead to recognition errors. This paper introduces two novel strategies—Scale Augmentation and Drop Attention—to address these hurdles.
Methodology
The proposed system utilizes a standard Encoder-Decoder architecture. The encoder is a modified ResNet-18 designed to extract high-resolution features, while the decoder is a Long Short-Term Memory (LSTM) network that generates LaTeX sequences.
Scale Augmentation
To combat the scale problem, the researchers implemented Scale Augmentation. Instead of normalizing images to a uniform height, training images are randomly scaled while maintaining their aspect ratio. These images are then zero-padded to a fixed large size ($256 \times 1024$). This technique forces the model to learn to recognize symbols regardless of their size, making it robust to the varying scales of bases and exponents.
Drop Attention
Drop Attention is a regularization module inspired by Dropout. During training, it randomly suppresses the peak features of the attention distribution. By preventing the model from relying solely on a single "perfect" feature point, Drop Attention encourages the network to utilize surrounding contextual information. This results in a more robust decoder that can handle imprecise attention maps during inference.
Experimental Results
The model was evaluated on the Competition on Recognition of Online Handwritten Mathematical Expressions (CROHME) datasets from 2014 and 2016. The results demonstrate that the combination of Scale Augmentation and Drop Attention significantly improves the Expression Recognition Rate (ExpRate).
| Dataset | Method | ExpRate (%) |
|---|---|---|
| CROHME 2014 | Previous SOTA (PAL-v2) | 48.88% |
| Ours (Single Model) | 56.59% | |
| Ours (Ensemble) | 60.45% | |
| CROHME 2016 | Previous SOTA (PAL-v2) | 49.61% |
| Ours (Single Model) | 54.58% | |
| Ours (Ensemble) | 58.06% |
The ensemble model, which combines five variations of the architecture, outperformed the 2016 competition winner by a substantial margin, setting a new state-of-the-art for offline HMER.
Conclusion
By addressing the fundamental spatial and scale-related challenges of mathematical notation, the proposed Scale Augmentation and Drop Attention techniques provide a significant boost to HMER performance. Future work will focus on extending this architecture to online HMER, incorporating temporal stroke data to further enhance recognition accuracy.
Summary updated