Transformer-Encoder and Decoder Models for Questions on Math

Anja Reusch, Maik Thiele, Wolfgang LehnerCLEF (Working Notes) 2022 · OpenReview

While Large Language Models (LLMs) have become remarkably adept at natural language tasks, mathematics remains a unique frontier. Mathematical formulas are not merely text; they are structured, symbolic representations requiring specialized processing. In their work for the ARQMath-3 lab, researchers Anja Reusch, Maik Thiele, and Wolfgang Lehner explore the fine-tuning of Transformer-based models to retrieve and generate answers for complex mathematical queries.

Methodology

The research team adopted a two-pronged approach to address the challenges of mathematical information retrieval (MIR) and answer generation.

The Encoder Pipeline (Retrieval)

For Task 1 (Answer Retrieval), the team compared three Transformer-Encoder architectures: BERT, ALBERT, and RoBERTa. To bridge the gap between natural language and symbolic math, they implemented a multi-stage training pipeline. This involved starting with a base model, performing domain-specific further pre-training on mathematical corpora, and finally fine-tuning on task-specific data.

The multi-stage training pipeline for mathematical Transformer models, progressing from general base models to domain-specific evaluation.

The Decoder Approach (Generation)

For Task 3 (Answer Generation), the team utilized GPT-2. The problem was framed as a text-completion task using the pattern: PROBLEM: [Question] SOLUTION: [Generated Answer]. They also experimented with "hints"—prefixing the solution with the word "HINT" to guide the model toward more concise and relevant outputs.

The Role of Specialized Tokenization

A core contribution of this work is the development of a customized tokenizer. Standard tokenizers often fragment LaTeX expressions into nonsensical sub-word units (e.g., breaking \frac into \, f, rac). The researchers added 501 specialized LaTeX tokens (such as \sum, \sqrt, and \frac) to the vocabulary to preserve the semantic integrity of formulas.

The tokenization process showing how individual characters are merged into meaningful LaTeX command tokens to preserve mathematical structure.

Datasets and Training

The models were trained on several large-scale datasets:

  • MathSE: 1 million questions and 1.4 million answers from Mathematics StackExchange (2010–2018).
  • AMPS (Auxiliary Mathematics Problems and Solutions): A 23GB dataset containing over 5 million problems from sources like Khan Academy.
  • Annotated Data: Judged test data from previous ARQMath competitions was used to refine the models, teaching them to distinguish between relevant and irrelevant mathematical arguments.

Key Results and Insights

Retrieval Performance

The best-performing model, an ALBERT-based variant named math_10, outperformed top models from previous years across all primary metrics, including nDCG', mAP', and p'@10. ALBERT proved particularly efficient, achieving high performance with fewer parameters due to its layer-sharing architecture.

Generation Feasibility

While generative math remains a difficult frontier, the GPT-2 models trained on the AMPS dataset demonstrated an ability to capture the topic of a question and generate syntactically correct LaTeX. However, the models occasionally suffered from a "copycat" problem, rephrasing the question rather than solving it, or "hallucinating" logically flawed mathematical steps.

Conclusion

The study demonstrates that "teaching" Transformers the specific structure of math formulas through specialized tokenization and domain-specific pre-training is essential for performance in MIR. While finding existing answers is now highly effective, generating novel, logically sound mathematical solutions remains a significant challenge for future research, likely requiring longer sequence lengths and more sophisticated logical verification methods.

Summary updated

BibTeX
@inproceedings{reusch2022transformer,
  author = {Anja Reusch and Maik Thiele and Wolfgang Lehner},
  title = {{Transformer-Encoder and Decoder Models for Questions on Math}},
  booktitle = {CLEF (Working Notes)},
  year = {2022},
  url = {https://openreview.net/forum?id=pCdvgmMvDC},
}
Figure