A Compact Model for Mathematics Problem Representations Distilled from BERT

Hao Ming, Xinguo Yu, Xiaotian Cheng, Zhenquan Shen, Xiaopan LyuAAAI 2025 · DOI

Abstract

Large Language Models (LLMs) like BERT have set benchmarks in Natural Language Processing but remain computationally expensive for specialized tasks like Math Word Problems (MWPs) in resource-constrained environments. This paper introduces a specialized knowledge distillation (KD) framework designed to create a compact math-solving model. By incorporating Part-of-Speech (POS) tagging in pre-training and utilizing bottleneck networks for intermediate feature distillation, the proposed model achieves approximately 95% of BERT's performance with only 10% of the parameters, significantly outperforming general-purpose compact models like TinyBERT.

Introduction

In the landscape of Natural Language Processing (NLP), Large Language Models (LLMs) are the standard for complex reasoning. However, Math Word Problems (MWPs)—which require mapping linguistic descriptions to mathematical equations—often do not require the full breadth of a general-purpose LLM. For intelligent tutoring systems on edge devices, these models are often too large and slow for practical deployment.

Research presented at AAAI-25, "A Compact Model for Mathematics Problem Representations Distilled from BERT," addresses this by developing a lightweight math solver. The resulting model is approximately 10.3 million parameters—roughly 10% the size of BERT—yet it retains the vast majority of its teacher's problem-solving accuracy.

The Challenge of Mathematical Distillation

General compact models like TinyBERT or DistilBERT are effective for sentiment analysis but struggle with the unique structural requirements of math. MWPs rely heavily on the relationships between symbols, quantities, and operations rather than general semantic meaning. Standard "task-agnostic" distillation often fails to capture this underlying logic. Furthermore, the significant "knowledge gap" between a high-dimensional teacher (e.g., BERT-base with 768 hidden units) and a low-dimensional student (e.g., 312 hidden units) makes direct imitation difficult.

Methodology: Deep Distillation for MWPs

The authors propose a Deep Distillation framework specifically tailored for mathematical representations. The process involves three distinct phases: pre-training, distillation, and fine-tuning.

Overview of the proposed training pipeline including pre-training, distillation, and fine-tuning.

Multi-Knowledge Distillation and Bottleneck Networks

To bridge the capacity gap, the student model learns not just from the teacher's final output but from its intermediate hidden states. Because the hidden dimensions differ, the researchers implemented linear bottleneck layers. These layers act as "translators," projecting the teacher's high-dimensional knowledge into a space the student can absorb effectively.

The Deep Distillation framework utilizing bottleneck layers and feature loss to bridge the gap between teacher and student models.

Tailored Pre-training

Before distillation, the student model undergoes specialized pre-training on two tasks:

  1. Masked Language Modeling (MLM): Focused on mathematical keywords and numerical values.
  2. Part-of-Speech (POS) Tagging: Identifying nouns (entities), verbs (operations), and quantities. This provides a structural foundation for understanding mathematical logic.

Key Insights

  • The Importance of Lower Layers: Analysis revealed that the first four layers of BERT account for 84.61% of its math-solving performance. This suggests that syntax and morphology—learned early in the network—are more critical for math than high-level abstract semantics.
  • POS as a Structural Proxy: POS tagging helps the model focus on the "skeleton" of a problem (e.g., recognizing that "total" implies addition) rather than getting distracted by specific entities (e.g., "apples" vs. "pears").
  • Uniform Mapping: Mapping student layers to teacher layers uniformly (e.g., student layer 1 to teacher layer 4) was found to be the most effective strategy for knowledge transfer, preventing the student from being overwhelmed by skipping too many steps of the teacher's logic.

Results

The compact model (3 layers, 312 hidden size) was evaluated against BERT and other compact models using the GTS decoder on the Math23k and Ape-clean datasets.

Model Parameters Math23k Acc. Ape-clean Acc.
BERT (Teacher) 109M 84.07% 84.33%
DistilBERT 67M 62.12% 75.71%
TinyBERT 14.5M 55.41% 72.98%
Proposed Model 10.3M 80.23% 80.46%

The results demonstrate that the proposed model is smaller than TinyBERT but achieves nearly 25% higher accuracy on Math23k, proving the efficacy of task-specific distillation.

Conclusion

By focusing on the specific linguistic and structural needs of mathematics, this research demonstrates that "less is more" when distillation is targeted. This specialized compact model enables high-performance math solving on low-power devices, making intelligent educational tools more accessible. Future work will look toward distilling knowledge from even larger generative models and expanding the approach to multi-lingual math corpora.

Summary updated

BibTeX
@inproceedings{ming2025compact,
  author = {Hao Ming and Xinguo Yu and Xiaotian Cheng and Zhenquan Shen and Xiaopan Lyu},
  title = {{A Compact Model for Mathematics Problem Representations Distilled from BERT}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.1609/aaai.v39i23.34669},
}
Figure