TL;DR
MathNet shifts the focus of Mathematical Expression Recognition (MER) from model architecture to data quality. By introducing a rigorous LaTeX normalization process and a multi-font dataset, the researchers achieved a massive performance leap, outperforming previous state-of-the-art models by up to 88.3% in error reduction.
What Problem Does This Paper Address?
Recognizing printed mathematical expressions (MER) and converting them into machine-readable LaTeX is notoriously difficult. Beyond the complex 2D structure of equations, the authors identify a "hidden" bottleneck: ground truth ambiguity.
In LaTeX, there are often dozens of ways to write the same visual formula. For example, \ge and \geq look identical but are different strings. Standard datasets like im2latex-100k are riddled with these inconsistencies, which "confuse" models during training. Furthermore, most existing benchmarks use only a single font, meaning models trained on them often fail when they encounter the diverse typography of real-world academic papers.
Methods
The authors propose a data-centric approach, arguing that better data is more effective than a more complex model. Their methodology consists of three pillars:
- Enhanced LaTeX Normalization: They developed a systematic pipeline to map any LaTeX string to a "canonical form." This involves removing redundant white space commands, standardizing curly bracket usage, ordering subscripts/superscripts consistently, and replacing synonymous tokens with a single standard.
- The MathNet Model: Eschewing the traditional CNN-LSTM architecture, they built MathNet. It uses a Convolutional Vision Transformer (CvT) as an encoder—which captures both local fine details and global structural relationships—paired with a standard Transformer decoder to handle long sequences of tokens.
- Strategic Augmentation: To make the model robust, they introduced random white pixel columns (to simulate spacing variations), blurring masks, and resolution changes during training.
Datasets
The paper utilizes four key datasets to prove its point:
- im2latex-100k: The classic benchmark, which the authors "cleaned" using their normalization.
- im2latexv2 (New): An upgraded version of the classic set featuring 30 different fonts for training and 59 for testing, ensuring the model doesn't just memorize one typeface.
- realFormula (New): A "reality check" dataset consisting of 121 manually annotated expressions extracted directly from arXiv papers.
- InftyMDB-1: A dataset of scanned mathematical expressions used to test robustness against real-world scanning noise.
Key Insights and Ideas
- Data Quality > Model Complexity: The most striking "aha moment" was that normalization alone provided a 12.2% improvement in Edit score. Refining the data was more impactful than the architectural shift to Transformers.
- The Resolution Gap: The researchers found that the common practice of using 100 DPI images for MER is insufficient. Moving to 600 DPI significantly boosted performance (from 78.2% to 98.0% Edit score), as higher resolution allows the model to distinguish between visually similar symbols like
,and.. - Font Generalization: By training on 30 fonts, the model became "font-agnostic." It performed nearly as well on 29 completely unseen fonts as it did on the training fonts, proving that diversity in data leads to true generalization.
Key Results
MathNet set new benchmarks across the board:
- im2latex-100k: Achieved a 94.7% Edit score, significantly higher than the previous SOTA (WYGIWYS at 88.6%).
- im2latexv2: Reached a staggering 97.2% Edit score, proving the effectiveness of the normalization on multi-font data.
- Real-World Performance: On the realFormula set, MathNet achieved an 88.3% Edit score, while the previous leading model (WYGIWYS) collapsed to just 27.5%.
- Noise Robustness: On the scanned InftyMDB-1 set, MathNet maintained an 89.2% Edit score, showing it can handle the "messiness" of physical documents.
Limitations and Future Work
Despite the gains, two main challenges remain:
- The "Array" Headache: Complex structures like matrices and aligned equations (the
arrayenvironment) still account for over 50% of the model's errors. The 2D logic of nested tables remains a hurdle for sequence-based decoders. - Multi-line Expressions: The current model is primarily optimized for single-line formulas. While a simple "y-cut" algorithm helps split multi-line blocks, a more integrated approach for multi-line recognition is needed.
- Mathematical Styles: Recognizing specific styles like
\mathbb(double-struck) or\mathcal(calligraphic) is still difficult because these styles are often inconsistent across different font packages.
The authors plan to integrate MathNet into a semi-automatic captioning system for PDFs to improve accessibility for visually impaired researchers.
Summary updated