Continual Pre-training of Language Models for Math Problem Understanding with Syntax-Aware Memory Network

Zheng Gong, Kun Zhou, Wayne Xin Zhao, Jing Sha, Shijin Wang, Ji-Rong Wen

Abstract

Understanding mathematical problems requires a deep integration of linguistic context and formulaic logic. Existing pre-trained language models (PLMs) often fail to capture the hierarchical structure of formulas, while graph-based models struggle to align these structures with textual descriptions. We introduce COMUS (Continual Pre-training for Math Problem Understanding with Syntax-Aware Memory Network), which bridges this gap by unifying text dependency trees and formula operator trees into a single Math Syntax Graph. A novel syntax-aware memory network facilitates fine-grained interaction between textual tokens and graph nodes. Experimental results demonstrate that COMUS significantly outperforms baseline models in tasks like knowledge point classification and question matching, particularly in data-scarce scenarios.

Introduction

Mathematical problems are inherently multi-modal, combining natural language with structured formulas. A primary challenge in automated math understanding is the "semantic gap": determining how specific textual descriptors (e.g., "acute angle") relate to specific variables or operators within a formula. While standard PLMs like BERT treat formulas as flat strings, they ignore the logical hierarchy essential for reasoning. Conversely, Graph Neural Networks (GNNs) effectively model formula structures but often lack the contextual depth of PLMs.

Methodology

COMUS addresses these challenges through two primary architectural innovations and a specialized pre-training regime.

Math Syntax Graph

To unify disparate representations, COMUS constructs a Math Syntax Graph. This structure merges a syntax dependency tree derived from the problem text with Operator Trees (OPTs) representing the formulas. These components are linked via a special [MATH] token, ensuring that the linguistic and logical flows are represented within a single, coherent graph.

Construction of the Math Syntax Graph from textual and formulaic inputs.

Syntax-Aware Memory Network

To fuse information between the PLM (text encoder) and the GAT (graph encoder), COMUS employs a Syntax-Aware Memory Network inserted between the top layers of the two models. This network utilizes two key mechanisms:

  • Multi-view Read: Text tokens query the math graph to retrieve relevant structural information.
  • Multi-view Write: Graph nodes update their representations based on the surrounding textual context.
The Syntax-Aware Memory Network architecture for fusing token and node representations.

Continual Pre-training

The model is pre-trained on a corpus of over one million high school math problems using three tasks:

  1. Masked Language Model (MLM): Standard token prediction adapted for mathematical symbols.
  2. Dependency Triplet Completion (DTC): Predicting missing relations in the syntax graph to enforce structural understanding.
  3. Text-Graph Contrastive Learning (TGCL): Aligning global text and graph representations.

Experiments and Results

COMUS was evaluated on four downstream tasks: Knowledge Point Classification (KPC), Question-Answer Matching (QAM), Question Relation Classification (QRC), and Similar Question Recommendation (SQR).

Key Findings

  • Superior Accuracy: In Knowledge Point Classification, COMUS achieved 72.6% accuracy, outperforming MathBERT (66.4%).
  • Few-Shot Robustness: In a stress test using only 5% of training data for KPC, COMUS maintained 22.1% accuracy, whereas MathBERT's performance dropped to 8.4%. This highlights the model's ability to capture the "essence" of math problems through its structural pre-training.
  • Ablation Studies: Removing the memory network or the DTC task led to significant performance degradation, confirming the importance of the fusion mechanism and structural awareness.

Conclusion

COMUS effectively bridges the gap between language and mathematical logic by explicitly modeling the structural dependencies of formulas and their textual context. While the model excels at classification and matching, future work will focus on extending its capabilities to numerical reasoning and step-by-step problem solving. Additionally, testing the framework on multi-lingual datasets remains a key objective.

Summary updated

BibTeX
@inproceedings{gong2022continual,
  author = {Zheng Gong and Kun Zhou and Wayne Xin Zhao and Jing Sha and Shijin Wang and Ji-Rong Wen},
  title = {{Continual Pre-training of Language Models for Math Problem Understanding with Syntax-Aware Memory Network}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2022},
  doi = {10.18653/v1/2022.acl-long.408},
}
Figure