Structural information in mathematical formulas for exercise difficulty prediction: a comparison of NLP representations

Ekaterina Loginova, Dries BenoitNAACL 2022

TL;DR

This paper explores how to represent mathematical formulas within educational exercises to better predict their difficulty. By treating formulas as structured data and using syntactic parse trees, the authors demonstrate that this approach significantly improves the accuracy of difficulty prediction models compared to simpler text-based methods. This work is crucial for developing more personalized and effective online learning systems.

What Problem Does This Paper Address?

Online learning platforms strive to offer personalized tutoring by recommending the most suitable learning activities for each student. A critical component of this personalization is accurately assessing the difficulty of learning content. While natural language processing (NLP) has made strides in representing text and computer vision for images, effectively representing mathematical formulas within educational contexts has remained a challenge. Formulas, with their unique syntax and symbols, require specialized methods beyond standard text processing to capture their inherent complexity and structure, which directly impacts exercise difficulty.

Methods

The core of this paper's contribution lies in its approach to representing mathematical formulas. Instead of treating formulas as simple strings of characters, the authors leverage structural embeddings. This involves parsing mathematical expressions into syntactic parse trees, similar to how sentences are parsed in NLP.

Specifically, they extract Abstract Syntax Trees (ASTs) or SymPy parse trees for formulas. The key idea is to represent each formula element (like an operator or a variable) by its path from the leaf node to the root of the parse tree. These paths, or "syntactic sequences," capture hierarchical information about the formula's structure. These sequences can then be fed into neural networks, such as LSTMs, to generate fixed-length vector representations.

The paper compares several ways of encoding this structural information:

  • Flat sequences: Concatenating the syntactic sequences into a single string.
  • Nested sequences: Using time-distributed wrappers to process the sequences directly.

These representations are then used as input to prediction models, either alone or combined with the textual description of the exercise.

Datasets

The experiments were conducted on two publicly available datasets:

  • DeepMind Mathematics Dataset: A synthetic dataset containing question-answer pairs labeled as "easy" or "hard." It offers a clear and unified format, making it suitable for comparing formula representations. The authors used a subset of 37,244 problems.
  • MATH Dataset: A dataset of 12,498 problems from US mathematics competitions, labeled with five difficulty levels. Formulas in this dataset are written in LaTeX.

Key Insights and Ideas

  • Formulas are more than just strings: The paper strongly suggests that the structural and syntactic properties of mathematical formulas are crucial for understanding their difficulty, and simple string-based representations are insufficient.
  • Hierarchical representations matter: Capturing the nested structure of formulas through parse trees provides richer information than flat or n-gram approaches.
  • Syntactic information complements text: Combining structural formula representations with textual descriptions of exercises leads to better difficulty prediction than using either component alone.
  • Domain-specific parsing is key: The authors developed custom scripts to handle the wide variety of notation conventions and parsing challenges specific to mathematical formulas, as existing tools had limited success.

Key Results

The paper evaluates its approach using 10-fold cross-validated ROC AUC.

  • Improvement over baselines: The proposed structural embeddings significantly outperform baseline models that rely solely on descriptions or raw formula text.
  • DeepMind Dataset: For the DeepMind dataset, combining textual descriptions with AST root paths (flat) improved ROC AUC from 0.69 to 0.73. Using nested syntactic sequences further boosted performance.
  • MATH Dataset: While not detailed as extensively as DeepMind in the results table, the paper indicates that structural embeddings also show promise for this dataset.
  • SymPy vs. AST: On the DeepMind dataset, SymPy root paths (flat) outperformed AST root paths, suggesting that the choice of parsing tool can influence results.

Limitations and Future Work

  • Limitations:
    • The paper acknowledges that the "word description dominates" the prediction when used alone, highlighting the need for effective formula representation.
  • The authors note that while their approach shows promise, it might be more impactful for more advanced tasks.
  • The parsing of formulas from raw text can be challenging due to notation variations, and their custom scripts achieved a 58% parsing correctness rate on the MATH dataset.
  • Future Work:
    • Apply the approach to more complex state exams.
  • Make the collected and processed dataset publicly available.
  • Predict the similarity of mathematical formulas by comparing their syntax trees.
  • Explore Graph Neural Networks (GNNs) for formula representation, as preliminary experiments showed improvement.

Summary updated

BibTeX
@inproceedings{loginova2022structural,
  author = {Ekaterina Loginova and Dries Benoit},
  title = {{Structural information in mathematical formulas for exercise difficulty prediction: a comparison of NLP representations}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2022},
}
Figure