Crohme2011: Competition on recognition of online handwritten mathematical expressions

Harold Mouchère, Utpal Garain, Jin Hyung Kim2015

TL;DR

This paper introduces CROHME2011, a competition designed to establish a benchmark for online handwritten mathematical expression recognition. Despite advancements in symbol recognition, the competition revealed that accurately recognizing entire mathematical expressions remains a significant challenge, with top systems achieving only around 20% accuracy.

What Problem Does This Paper Address?

Recognizing handwritten mathematical expressions is a notoriously difficult problem in pattern recognition. Unlike standard text, mathematical expressions have a complex, two-dimensional structure with inherent ambiguities in symbol relationships and parsing. This complexity makes it hard to compare different research approaches because they often use proprietary datasets and evaluation methods. The CROHME2011 competition aimed to address this by creating a standardized platform with a shared dataset and evaluation metrics, allowing for a clear assessment of the state-of-the-art and identifying key challenges.

Methods

The CROHME2011 competition involved several research groups submitting their systems for recognizing online handwritten mathematical expressions. The core methodology involved:

  • Standardized Data Format: An InkML format was used to store expression data, including stroke information, symbol-level ground truth, and expression-level ground truth (in MathML format).
  • Hierarchical Grammars: Two levels of grammars (Grammar-I and Grammar-II) were defined to categorize expressions by complexity, allowing for a more nuanced evaluation. Grammar-I represented simpler expressions, while Grammar-II allowed for more complex structures and recursions.
  • Shared Training and Test Datasets: A training set of 921 expressions and a test set of 348 expressions were provided.
  • Multi-faceted Evaluation: Systems were evaluated on four metrics: stroke-level classification rate (STrec), symbol segmentation rate (SYMseg), symbol recognition rate (SYMrec), and expression-level recognition rate (EXPrec).
  • Final Rating: The ultimate ranking was based on the expression-level recognition accuracy (EXPrec).

The participating systems employed various techniques, including Hidden Markov Models (HMMs), stochastic grammars, and parsing algorithms, often combining online and offline information.

Datasets

  • Training Dataset: 921 online handwritten mathematical expressions. This dataset was further divided into Part-I (simpler expressions) and Part-II (more complex expressions).
  • Test Dataset: 348 online handwritten mathematical expressions, also divided into Part-I and Part-II.
  • New Benchmark: The competition itself established a new benchmark for evaluating mathematical expression recognition systems, providing a standardized dataset and evaluation methodology that was previously lacking.

Key Insights and Ideas

  • The "Expression Level" is the Bottleneck: While systems showed decent performance in recognizing individual strokes and symbols, accurately reconstructing the entire mathematical expression proved to be the most significant hurdle. This highlights the critical importance of understanding the complex 2D structure and relationships between symbols.
  • Ambiguity is Pervasive: The inherent ambiguities in handwritten mathematical notation, especially in its 2D layout, are the primary reason for the low expression-level recognition accuracies.
  • Need for Standardization: The competition underscored the critical need for shared datasets and evaluation protocols to truly gauge progress in this research area. Without them, it's difficult to compare different approaches objectively.
  • Baseline Performance is Surprisingly Low: Even a baseline system developed by the organizers achieved only 22.41% expression-level accuracy on the test data, indicating the difficulty of the task.

Key Results

The competition revealed a significant gap between symbol-level recognition and expression-level recognition:

  • Expression-Level Accuracy: The best performing system (System-III) achieved an expression-level recognition accuracy (EXPrec) of 19.83% on the Part-II test data.
  • Baseline Performance: A baseline system achieved 22.41% EXPrec on the same dataset.
  • Symbol Recognition: In contrast, symbol recognition rates (SYMrec) were much higher, with System-III achieving 92.22% on Part-I and 92.56% on Part-II.
  • Consistency Across Datasets: Systems showed consistent performance between the Part-I and Part-II test datasets, though EXPrec generally degraded on the more complex Part-II expressions.

Limitations and Future Work

  • Low Expression-Level Accuracy: The most significant limitation highlighted is the very low expression-level recognition accuracy across all participating systems. This indicates that current methods are far from solving the problem comprehensively.
  • Future Research Direction: The paper clearly points to the need for further research into understanding and resolving the ambiguities inherent in the 2D structure of mathematical expressions.
  • Continued Competition: The authors suggest that future iterations of this competition are necessary to attract more research groups and foster the development of new recognition methods.
  • Dataset Availability: The datasets and evaluation results are made available, encouraging researchers to benchmark their own systems and contribute to the field's progress.

Summary updated

Abstract

A competition on recognition of online handwritten mathematical expressions is organized. Recognition of mathematical expressions has been an attractive problem for the pattern recognition community because of the presence of enormous uncertainties and ambiguities as encountered during parsing of the two-dimensional structure of expressions. The goal of this competition is to bring out a state of the art for the related research. Three labs come together to organize the event and six other research groups participated the competition. The competition defines a standard format for presenting information, provides a training set of 921 expressions and supplies the underlying grammar for understanding the content of the training data. Participants were invited to submit their recognizers which were tested with a new set of 348 expressions. Systems are evaluated based on four different aspects of the recognition problem. However, the final rating of the systems is done based on their correct expression recognition accuracies. The best expression level recognition accuracy (on the test data) shown by the competing systems is 19.83 % whereas a baseline system developed by one of the organizing groups reports an accuracy 22.41 % on the same dataset. Keywords-Evaluation; Mathematical expressions; Online handwriting, symbol recognition.
BibTeX
@misc{mouchere2015crohme,
  author = {Harold Mouchère and Harold Mouchère and Utpal Garain and Jin Hyung Kim and Jin Hyung Kim},
  title = {{Crohme2011: Competition on recognition of online handwritten mathematical expressions}},
  year = {2015},
}
Figure