Attention Guidance Mechanism for Handwritten Mathematical Expression Recognition

Yutian Liu, Wenjun Ke, Jianguo WeiarXiv 2024 · DOI · arXiv

Introduction

Handwritten Mathematical Expression Recognition (HMER) presents a unique challenge in computer vision. Unlike linear text, mathematical formulas are two-dimensional structures containing exponents, subscripts, and nested fractions. While modern encoder-decoder models using attention mechanisms have made significant strides, they often suffer from "context leakage"—a phenomenon where the model prematurely attends to future symbols, leading to under-parsing and missing elements in the final LaTeX output.

The Challenge of Context Leakage

Standard HMER models often employ a "coverage mechanism" to prevent over-parsing (repeating symbols). However, coverage mechanisms are inherently backward-looking; they track what has already been processed. They cannot prevent the model from "leaking" attention into future regions of the image. When a model mistakenly focuses on a future symbol (e.g., a superscript later in the equation), it may skip the current target symbol entirely because it perceives the future symbol as already "covered."

Proposed Methodology: Attention Guidance Mechanism (AGM)

To address context leakage, researchers from Tianjin University proposed the Attention Guidance Mechanism (AGM), integrated into the Transformer-based CoMER model. AGM consists of two primary strategies:

1. Self-Guidance

In multi-head attention, different heads observe the image from various feature subspaces. While their perspectives differ, they should ideally converge on the same physical symbol. Self-guidance enforces a "consensus" across these heads, using a combined attention map to suppress noisy or inconsistent activations that might lead to leakage.

2. Neighbor-Guidance

The researchers observed that in deep Transformer decoders, alignment is a multi-stage process. Middle layers often do not focus on the current symbol but rather on the previously decoded "neighbor." Neighbor-guidance leverages this by using the final attention map from the previous decoding step to "prime" the middle layers of the current step. This ensures a logical progression of focus across the image.

The Neighbor-Guidance mechanism facilitates information propagation across decoder layers and time steps, ensuring the model maintains a consistent focus on the sequence of symbols.

Experimental Evaluation

The AGM-enhanced model, "AttnGuide," was evaluated on the CROHME 2014, 2016, and 2019 benchmarks. These datasets represent the standard for HMER, featuring a wide variety of handwritten mathematical notation.

Performance Metrics

AttnGuide achieved new state-of-the-art (SOTA) results across all three benchmarks:

  • CROHME 2014: 60.75% ExpRate (Expression Recognition Rate)
  • CROHME 2016: 61.99% ExpRate
  • CROHME 2019: 63.30% ExpRate

These results demonstrate a significant improvement over previous models, particularly in reducing under-parsing errors.

Qualitative Analysis

Visualizing the attention heatmaps reveals how the guidance mechanism stabilizes the model's focus. By suppressing erroneous future-leaning activations, the model accurately tracks the sequence of symbols even in complex, multi-part expressions.

Visualization of attention heatmaps for specific tokens (e.g., \frac, j, exponents). The AGM ensures that the model's attention is precisely aligned with the target symbol at each decoding step.

Conclusion

The Attention Guidance Mechanism provides a robust solution to the problem of context leakage in HMER. By utilizing self-guidance for consensus and neighbor-guidance for temporal consistency, the model achieves superior alignment and recognition accuracy. This approach highlights the importance of guiding attention not just based on past coverage, but through inter-layer and inter-head coordination.

Reference: Liu, Y., Ke, W., & Wei, J. (2024). Attention Guidance Mechanism for Handwritten Mathematical Expression Recognition. arXiv:2403.01756v2 [cs.CV].

Summary updated

Abstract

Handwritten mathematical expression recognition (HMER) is challenging in image-to-text tasks due to the complex layouts of mathematical expressions and suffers from problems including over-parsing and under-parsing. To solve these, previous HMER methods improve the attention mechanism by utilizing historical alignment information. However, this approach has limitations in addressing under-parsing since it cannot correct the erroneous attention on image areas that should be parsed at subsequent decoding steps. This faulty attention causes the attention module to incorporate future context into the current decoding step, thereby confusing the alignment process. To address this issue, we propose an attention guidance mechanism to explicitly suppress attention weights in irrelevant areas and enhance the appropriate ones, thereby inhibiting access to information outside the intended context. Depending on the type of attention guidance, we devise two complementary approaches to refine attention weights: self-guidance that coordinates attention of multiple heads and neighbor-guidance that integrates attention from adjacent time steps. Experiments show that our method outperforms existing state-of-the-art methods, achieving expression recognition rates of 60.75% / 61.81% / 63.30% on the CROHME 2014/ 2016/ 2019 datasets.
BibTeX
@misc{liu2024attention,
  author = {Yutian Liu and Wenjun Ke and Jianguo Wei},
  title = {{Attention Guidance Mechanism for Handwritten Mathematical Expression Recognition}},
  year = {2024},
  eprint = {2403.01756},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2403.01756},
}
Figure