MathNet: A Data-Centric Approach for Printed Mathematical Expression Recognition

Felix M. Schmitt-Koopmann, Elaine M. Huang, Hans-Peter Hutter, Thilo Stadelmann, Alireza Darvishy

Abstract

Printed mathematical expression recognition (MER) models are usually trained and tested using LaTeX-generated mathematical expressions (MEs) as input and the LaTeX source code as ground truth. As the same ME can be generated by various different LaTeX source codes, this leads to unwanted variations in the ground truth data that bias test performance results and hinder efficient learning. In addition, the use of only one font to generate the MEs heavily limits the generalization of the reported results to realistic scenarios. We propose a data-centric approach to overcome this problem, and present convincing experimental results: Our main contribution is an enhanced LaTeX normalization to map any LaTeX ME to a canonical form. Based on this process, we developed an improved version of the benchmark dataset im2latex-100k, featuring 30 fonts instead of one. Second, we introduce the real-world dataset realFormula, with MEs extracted from papers. Third, we developed a MER model, MathNet, based on a convolutional vision transformer, with superior results on all four test sets (im2latex-100k, im2latexv2, realFormula, and InftyMDB-1), outperforming the previous state of the art by up to 88.3%.

Introduction

Recognizing printed mathematical expressions (MER) and converting them into machine-readable LaTeX is a complex task due to the two-dimensional structure of equations. However, a significant yet often overlooked bottleneck is ground truth ambiguity. In LaTeX, a single visual formula can be represented by numerous different strings. For instance, \ge and \geq produce identical symbols but are treated as distinct targets by models. This inconsistency "confuses" models during training and leads to misleading performance metrics. Furthermore, standard benchmarks like im2latex-100k typically use only a single font, which fails to prepare models for the typographic diversity found in real-world academic literature.

Data-Centric Methodology

MathNet addresses these issues through a data-centric approach, prioritizing data quality and diversity over architectural complexity alone.

Enhanced LaTeX Normalization

The researchers developed a systematic pipeline to map any LaTeX string to a "canonical form." This process involves standardizing curly bracket usage, ordering subscripts and superscripts consistently, removing redundant spacing commands, and replacing synonymous tokens with a single standard.

LaTeX normalization maps synonymous strings to a canonical form, reducing ground truth ambiguity.

Multi-Font and Real-World Datasets

To improve generalization, the authors introduced im2latexv2, an upgraded version of the classic benchmark featuring 30 different fonts for training and 59 for testing. They also created realFormula, a "reality check" dataset consisting of 121 manually annotated expressions extracted directly from arXiv papers, and utilized InftyMDB-1 to test robustness against scanned document noise.

The MathNet Architecture

While data was the primary focus, the authors also introduced MathNet, a model designed to leverage high-quality data. Moving away from traditional CNN-LSTM architectures, MathNet utilizes a Convolutional Vision Transformer (CvT) as an encoder. This choice allows the model to capture both local fine details (crucial for distinguishing similar symbols) and global structural relationships. The encoder is paired with a standard Transformer decoder to manage long token sequences.

The MathNet architecture uses a Convolutional Vision Transformer (CvT) encoder to capture both local and global features.

Key Insights and Results

The experimental results underscore the power of data-centric refinement:

  • Data Quality Impact: Normalization alone provided a 12.2% improvement in Edit score, proving more impactful than the shift to Transformer architectures.
  • Resolution Matters: Increasing image resolution from 100 DPI to 600 DPI boosted the Edit score from 78.2% to 98.0%, as it allowed the model to distinguish between visually similar symbols like commas and periods.
  • Superior Performance: MathNet achieved a 94.7% Edit score on im2latex-100k (surpassing the previous SOTA of 88.6%) and a staggering 97.2% on the multi-font im2latexv2.
  • Real-World Generalization: On the realFormula set, MathNet maintained an 88.3% Edit score, while the previous leading model (WYGIWYS) dropped to 27.5%.

Conclusion

MathNet demonstrates that addressing data inconsistencies and font diversity is critical for advancing MER. By training on 30 fonts, the model became effectively "font-agnostic," performing nearly as well on unseen fonts as on training ones. Future work will focus on handling complex 2D structures like matrices and multi-line expressions, which remain the primary sources of error.

Summary updated

Abstract

Printed mathematical expression recognition (MER) models are usually trained and tested using LaTeX-generated mathematical expressions (MEs) as input and the LaTeX source code as ground truth. As the same ME can be generated by various different LaTeX source codes, this leads to unwanted variations in the ground truth data that bias test performance results and hinder efficient learning. In addition, the use of only one font to generate the MEs heavily limits the generalization of the reported results to realistic scenarios. We propose a data-centric approach to overcome this problem, and present convincing experimental results: Our main contribution is an enhanced LaTeX normalization to map any LaTeX ME to a canonical form. Based on this process, we developed an improved version of the benchmark dataset im2latex-100k, featuring 30 fonts instead of one. Second, we introduce the real-world dataset realFormula, with MEs extracted from papers. Third, we developed a MER model, MathNet, based on a convolutional vision transformer, with superior results on all four test sets (im2latex-100k, im2latexv2, realFormula, and InftyMDB-1), outperforming the previous state of the art by up to 88.3%.
BibTeX
@inproceedings{schmittkoopmann2024mathnet,
  author = {Felix M. Schmitt-Koopmann and Elaine M. Huang and Hans-Peter Hutter and Thilo Stadelmann and Alireza Darvishy},
  title = {{MathNet: A Data-Centric Approach for Printed Mathematical Expression Recognition}},
  booktitle = {IEEE Access},
  year = {2024},
  url = {https://openreview.net/forum?id=KJ24cWXMcu},
}
Figure