Abstract
Large Language Models (LLMs) like BERT have set benchmarks in Natural Language Processing but remain computationally expensive for specialized tasks like Math Word Problems (MWPs) in resource-constrained environments. This paper introduces a specialized knowledge distillation (KD) framework designed to create a compact math-solving model. By incorporating Part-of-Speech (POS) tagging in pre-training and utilizing bottleneck networks for intermediate feature distillation, the proposed model achieves approximately 95% of BERT's performance with only 10% of the parameters, significantly outperforming general-purpose compact models like TinyBERT.
Introduction
In the landscape of Natural Language Processing (NLP), Large Language Models (LLMs) are the standard for complex reasoning. However, Math Word Problems (MWPs)—which require mapping linguistic descriptions to mathematical equations—often do not require the full breadth of a general-purpose LLM. For intelligent tutoring systems on edge devices, these models are often too large and slow for practical deployment.
Research presented at AAAI-25, "A Compact Model for Mathematics Problem Representations Distilled from BERT," addresses this by developing a lightweight math solver. The resulting model is approximately 10.3 million parameters—roughly 10% the size of BERT—yet it retains the vast majority of its teacher's problem-solving accuracy.
The Challenge of Mathematical Distillation
General compact models like TinyBERT or DistilBERT are effective for sentiment analysis but struggle with the unique structural requirements of math. MWPs rely heavily on the relationships between symbols, quantities, and operations rather than general semantic meaning. Standard "task-agnostic" distillation often fails to capture this underlying logic. Furthermore, the significant "knowledge gap" between a high-dimensional teacher (e.g., BERT-base with 768 hidden units) and a low-dimensional student (e.g., 312 hidden units) makes direct imitation difficult.
Methodology: Deep Distillation for MWPs
The authors propose a Deep Distillation framework specifically tailored for mathematical representations. The process involves three distinct phases: pre-training, distillation, and fine-tuning.
Multi-Knowledge Distillation and Bottleneck Networks
To bridge the capacity gap, the student model learns not just from the teacher's final output but from its intermediate hidden states. Because the hidden dimensions differ, the researchers implemented linear bottleneck layers. These layers act as "translators," projecting the teacher's high-dimensional knowledge into a space the student can absorb effectively.
Tailored Pre-training
Before distillation, the student model undergoes specialized pre-training on two tasks:
- Masked Language Modeling (MLM): Focused on mathematical keywords and numerical values.
- Part-of-Speech (POS) Tagging: Identifying nouns (entities), verbs (operations), and quantities. This provides a structural foundation for understanding mathematical logic.
Key Insights
- The Importance of Lower Layers: Analysis revealed that the first four layers of BERT account for 84.61% of its math-solving performance. This suggests that syntax and morphology—learned early in the network—are more critical for math than high-level abstract semantics.
- POS as a Structural Proxy: POS tagging helps the model focus on the "skeleton" of a problem (e.g., recognizing that "total" implies addition) rather than getting distracted by specific entities (e.g., "apples" vs. "pears").
- Uniform Mapping: Mapping student layers to teacher layers uniformly (e.g., student layer 1 to teacher layer 4) was found to be the most effective strategy for knowledge transfer, preventing the student from being overwhelmed by skipping too many steps of the teacher's logic.
Results
The compact model (3 layers, 312 hidden size) was evaluated against BERT and other compact models using the GTS decoder on the Math23k and Ape-clean datasets.
| Model | Parameters | Math23k Acc. | Ape-clean Acc. |
|---|---|---|---|
| BERT (Teacher) | 109M | 84.07% | 84.33% |
| DistilBERT | 67M | 62.12% | 75.71% |
| TinyBERT | 14.5M | 55.41% | 72.98% |
| Proposed Model | 10.3M | 80.23% | 80.46% |
The results demonstrate that the proposed model is smaller than TinyBERT but achieves nearly 25% higher accuracy on Math23k, proving the efficacy of task-specific distillation.
Conclusion
By focusing on the specific linguistic and structural needs of mathematics, this research demonstrates that "less is more" when distillation is targeted. This specialized compact model enables high-performance math solving on low-power devices, making intelligent educational tools more accessible. Future work will look toward distilling knowledge from even larger generative models and expanding the approach to multi-lingual math corpora.
Summary updated