Introduction
Handwritten Mathematical Expression Recognition (HMER) presents a unique challenge in computer vision. Unlike linear text, mathematical formulas are two-dimensional structures containing exponents, subscripts, and nested fractions. While modern encoder-decoder models using attention mechanisms have made significant strides, they often suffer from "context leakage"—a phenomenon where the model prematurely attends to future symbols, leading to under-parsing and missing elements in the final LaTeX output.
The Challenge of Context Leakage
Standard HMER models often employ a "coverage mechanism" to prevent over-parsing (repeating symbols). However, coverage mechanisms are inherently backward-looking; they track what has already been processed. They cannot prevent the model from "leaking" attention into future regions of the image. When a model mistakenly focuses on a future symbol (e.g., a superscript later in the equation), it may skip the current target symbol entirely because it perceives the future symbol as already "covered."
Proposed Methodology: Attention Guidance Mechanism (AGM)
To address context leakage, researchers from Tianjin University proposed the Attention Guidance Mechanism (AGM), integrated into the Transformer-based CoMER model. AGM consists of two primary strategies:
1. Self-Guidance
In multi-head attention, different heads observe the image from various feature subspaces. While their perspectives differ, they should ideally converge on the same physical symbol. Self-guidance enforces a "consensus" across these heads, using a combined attention map to suppress noisy or inconsistent activations that might lead to leakage.
2. Neighbor-Guidance
The researchers observed that in deep Transformer decoders, alignment is a multi-stage process. Middle layers often do not focus on the current symbol but rather on the previously decoded "neighbor." Neighbor-guidance leverages this by using the final attention map from the previous decoding step to "prime" the middle layers of the current step. This ensures a logical progression of focus across the image.
Experimental Evaluation
The AGM-enhanced model, "AttnGuide," was evaluated on the CROHME 2014, 2016, and 2019 benchmarks. These datasets represent the standard for HMER, featuring a wide variety of handwritten mathematical notation.
Performance Metrics
AttnGuide achieved new state-of-the-art (SOTA) results across all three benchmarks:
- CROHME 2014: 60.75% ExpRate (Expression Recognition Rate)
- CROHME 2016: 61.99% ExpRate
- CROHME 2019: 63.30% ExpRate
These results demonstrate a significant improvement over previous models, particularly in reducing under-parsing errors.
Qualitative Analysis
Visualizing the attention heatmaps reveals how the guidance mechanism stabilizes the model's focus. By suppressing erroneous future-leaning activations, the model accurately tracks the sequence of symbols even in complex, multi-part expressions.
Conclusion
The Attention Guidance Mechanism provides a robust solution to the problem of context leakage in HMER. By utilizing self-guidance for consensus and neighbor-guidance for temporal consistency, the model achieves superior alignment and recognition accuracy. This approach highlights the importance of guiding attention not just based on past coverage, but through inter-layer and inter-head coordination.
Reference: Liu, Y., Ke, W., & Wei, J. (2024). Attention Guidance Mechanism for Handwritten Mathematical Expression Recognition. arXiv:2403.01756v2 [cs.CV].
Summary updated