TL;DR
This paper introduces an end-to-end graph-to-graph framework for Online Handwritten Mathematical Expression Recognition (HMER) that bypasses traditional text-generation decoders. By modeling handwritten strokes as nodes and their spatial-temporal relationships as edges, the proposed Edge-weighted Graph Attention Network (EGAT) simultaneously classifies symbols and parses layout structures using both local and global context.
What Problem Does This Paper Address?
Recognizing handwritten mathematical expressions is uniquely challenging because it requires understanding both the identity of individual symbols and their complex 2D spatial arrangements (such as superscripts, fractions, and matrices). Most modern deep learning approaches treat this as a sequence-to-sequence problem, using an encoder-decoder architecture to translate stroke trajectories or images into LaTeX strings.
However, these sequence-to-sequence models operate in a latent space and fail to directly leverage the explicit graph structure inherent in mathematical layouts. This often leads to structural errors and a lack of direct alignment between input strokes and output symbols. The authors tackle this gap by proposing a method that directly maps input strokes to a structured layout graph.
Methods
The authors propose a Graph-to-Graph transformation approach that treats HMER as a simultaneous node and edge classification task:
- Graph Representation: Individual strokes are treated as nodes, and potential relationships between them are edges. Node features capture normalized stroke coordinates, while edge features capture spatial relationships using a Fuzzy Relative Positioning Template (FRPT) along with distance metrics.
- Edge-weighted Graph Attention Mechanism (EGAT): Unlike standard Graph Attention Networks (GATs) that treat edges as simple scalar weights, EGAT dynamically updates both node and edge feature representations. It introduces a Message Concatenate strategy where node updates incorporate edge features, and edge updates incorporate the features of their connected nodes.
- Local & Global Graph Modeling: To capture both fine-grained and holistic context, the authors combine Local Graph Modeling (LGM)—which builds connectivity using Line-of-Sight (LOS) visibility and temporal neighbors—with Global Graph Modeling (GGM). GGM introduces a virtual master node connected to all other nodes to aggregate expression-wide context.
- Optimization: The network uses residual connections, intermediate supervision via auxiliary readouts, and a multi-objective loss function combining Cross-Entropy for nodes and Focal Loss to handle heavily imbalanced edge classes.
Datasets
The model was trained and evaluated on the following benchmarks:
- CROHME 2023: The primary dataset used for ablation studies and evaluation, featuring large-scale online handwritten mathematical expressions with stroke-level annotations.
- CROHME 2016 & CROHME 2019: Used to benchmark the proposed system against existing state-of-the-art online and offline methods.
Key Insights and Ideas
- Edges as First-Class Citizens: In mathematical expressions, the relationship between symbols (e.g., "above" vs. "superscript") is just as important as the symbol identity. Treating edges as dynamic feature vectors rather than static weights allows the network to learn complex spatial syntax effectively.
- The Master Node Guides Ambiguity: Heatmap visualizations reveal that the global master node successfully passes information to structurally ambiguous characters. For example, it helps the network differentiate between a capital 'C' and a lowercase 'c' by providing the context and scale of the entire expression.
- Temporal Ordering Prevents Directional Confusion: By forcing edges to point from earlier-written strokes to later-written strokes (via an Edited Stroke Label Graph), the model avoids the confusion of bidirectional spatial relations while preserving the layout structure.
Key Results
- State-of-the-Art Performance: On the CROHME 2019 dataset, the Global Graph Modeling approach (GGM-EGAT) achieved an expression recognition accuracy of 60.72%, outperforming the previous best graph-to-graph model (G2G at 52.05%) and offline graph methods (GETD at 54.13%).
- CROHME 2016: GGM-EGAT achieved 56.67% accuracy, setting a new benchmark among models trained without external data or pre-training.
- Ablation Validation: Ablation studies proved that the Message Concatenate strategy boosted primitive edge classification accuracy significantly. Combining all strategies yielded 94.40% node accuracy and 97.50% edge accuracy on the CROHME 2023 validation set.
Limitations and Future Work
- Lack of Structural Constraints: Because edges are predicted independently, the model can still output structurally invalid trees (e.g., missing a basic "Right" relation between valid symbols) due to a lack of global grammatical constraints during the final decision process.
- Future Directions: The authors plan to explore a "Stroke to Sequence" strategy to handle larger datasets, investigate "Graph to Stroke" synthesis for generating handwritten math, and adapt the framework to other 2D structured domains like chemical notation and musical score analysis.
Summary updated