Abstract: We present a new visual parsing method based on convolutional neural networks for handwritten mathematical formulas. The Query-Driven Global Graph Attention (QDGGA) parsing model employs multi-task learning, and uses a single feature representation for locating, classifying, and relating symbols. First, a Line-Of-Sight (LOS) graph is computed over the handwritten strokes in a formula. Second, class distributions for LOS nodes and edges are obtained using query-specific feature filters (i.e., attention) in a single feed-forward pass. Finally, a Maximum Spanning Tree (MST) is extracted from the weighted graph. Our preliminary results show that this is a promising new approach for visual parsing of handwritten formulas. Our data and source code are publicly available.
Introduction
Handwritten mathematical expression recognition (HMER) is inherently difficult because mathematical notation is non-linear and two-dimensional. Unlike standard text, math involves complex spatial relationships such as exponents, subscripts, and fractions. While current state-of-the-art models often use Encoder-Decoder architectures to flatten these 2D structures into 1D sequences (like LaTeX strings), this approach often requires complex attention mechanisms to maintain spatial context. This paper introduces QD-GGA, a graph-based visual parser that learns structure directly from visual features without the need for a formal grammar.
Methodology
The QD-GGA framework treats HMER as a graph parsing problem, operating in several distinct stages:
Graph Construction and Feature Extraction
Rather than using a computationally expensive complete graph where every stroke is connected to every other stroke, the model constructs a Line-of-Sight (LOS) graph. This prunes the search space by only connecting strokes that are geometrically visible to one another, reducing edge complexity from $O(n^2)$ to approximately $3.3n$. Simultaneously, an SE-ResNext CNN backbone processes the input image to generate a global feature map.
Query-Driven Global Graph Attention
The core innovation is the attention mechanism. The model uses binary masks representing specific nodes (strokes) or edges (stroke pairs) as queries. These queries are convolved into "relevance maps" that filter the global feature map. This allows the model to extract specific features for any part of the graph in a single feed-forward pass. The authors noted that using trainable "soft" attention masks instead of static binary masks improved recognition rates by roughly 3%.
Multi-Task Learning and Tree Extraction
The model is trained to perform three tasks simultaneously: symbol classification, symbol detection (segmentation), and relationship classification.
Finally, the system extracts a Symbol Layout Tree (SLT) by applying Edmond’s arborescence algorithm to find the Maximum Spanning Tree (MST) from the weighted adjacency matrix produced by the network.
Results and Discussion
The model was evaluated using the CROHME 2014, 2016, and 2019 benchmarks.
- Accuracy: QD-GGA achieved expression recognition rates of 32.04% and 32.84% on the 2014 and 2016 datasets, respectively. Jointly training the segmentation, classification, and relationship modules provided a 5% boost in accuracy compared to training them in isolation.
- Efficiency: By avoiding the step-by-step nature of recurrent decoders, QD-GGA is significantly faster. It processed the CROHME 2014 dataset in 59 seconds, compared to 377 seconds for the competing TAP model.
While the accuracy currently trails top-performing sequence-to-sequence models (which reach 48-54%), these preliminary results demonstrate a much faster, grammar-free architecture. Future work will focus on incorporating temporal stroke data and improving robustness to variations in character scale.
Summary updated