Improving Attention-Based Handwritten Mathematical Expression Recognition with Scale Augmentation and Drop Attention

Zhe Li, Lianwen Jin, Songxuan Lai, Yecheng Zhu

Introduction

Handwritten Mathematical Expression Recognition (HMER) presents a unique challenge compared to standard optical character recognition (OCR). While text is typically linear, mathematical expressions are inherently two-dimensional, featuring complex spatial relationships such as superscripts, subscripts, fractions, and nested radicals.

Examples of handwritten mathematical expressions showing varying complexity and spatial arrangements.

Current attention-based models often struggle with two specific issues: the "Scale Problem," where fixed-height normalization obscures small symbols like exponents, and "Attention Imprecision," where slight misalignments in the model's focus lead to recognition errors. This paper introduces two novel strategies—Scale Augmentation and Drop Attention—to address these hurdles.

Methodology

The proposed system utilizes a standard Encoder-Decoder architecture. The encoder is a modified ResNet-18 designed to extract high-resolution features, while the decoder is a Long Short-Term Memory (LSTM) network that generates LaTeX sequences.

The attention-based encoder-decoder architecture for HMER, showing the flow from handwritten image to LaTeX output with attention maps.

Scale Augmentation

To combat the scale problem, the researchers implemented Scale Augmentation. Instead of normalizing images to a uniform height, training images are randomly scaled while maintaining their aspect ratio. These images are then zero-padded to a fixed large size ($256 \times 1024$). This technique forces the model to learn to recognize symbols regardless of their size, making it robust to the varying scales of bases and exponents.

Drop Attention

Drop Attention is a regularization module inspired by Dropout. During training, it randomly suppresses the peak features of the attention distribution. By preventing the model from relying solely on a single "perfect" feature point, Drop Attention encourages the network to utilize surrounding contextual information. This results in a more robust decoder that can handle imprecise attention maps during inference.

Experimental Results

The model was evaluated on the Competition on Recognition of Online Handwritten Mathematical Expressions (CROHME) datasets from 2014 and 2016. The results demonstrate that the combination of Scale Augmentation and Drop Attention significantly improves the Expression Recognition Rate (ExpRate).

Dataset Method ExpRate (%)
CROHME 2014 Previous SOTA (PAL-v2) 48.88%
Ours (Single Model) 56.59%
Ours (Ensemble) 60.45%
CROHME 2016 Previous SOTA (PAL-v2) 49.61%
Ours (Single Model) 54.58%
Ours (Ensemble) 58.06%

The ensemble model, which combines five variations of the architecture, outperformed the 2016 competition winner by a substantial margin, setting a new state-of-the-art for offline HMER.

Conclusion

By addressing the fundamental spatial and scale-related challenges of mathematical notation, the proposed Scale Augmentation and Drop Attention techniques provide a significant boost to HMER performance. Future work will focus on extending this architecture to online HMER, incorporating temporal stroke data to further enhance recognition accuracy.

Summary updated

BibTeX
@inproceedings{li2020improving,
  author = {Zhe Li and Lianwen Jin and Songxuan Lai and Yecheng Zhu},
  title = {{Improving Attention-Based Handwritten Mathematical Expression Recognition with Scale Augmentation and Drop Attention}},
  booktitle = {ICFHR},
  year = {2020},
  doi = {10.1109/icfhr2020.2020.00041},
}
Figure