Abstract
Addressing the challenge of high annotation costs in solving Math Word Problems (MWPs) through full supervision with intermediate equations, recent works have proposed weakly supervised task settings that rely solely on the final answer as a supervised signal. Existing leading approaches typically employ various search techniques to infer intermediate equations, but cannot ensure their semantic consistency with natural language descriptions. The rise of Large Language Models (LLMs) like ChatGPT has opened up new possibilities for addressing MWPs directly. However, the computational demands of LLMs make them less than ideal for use in settings where resources are tight. In light of these challenges, we introduce an innovative two-stage framework that adeptly transfers mathematical Expertise from large to tiny language models. In the Distillation Stage, we propose a series of extraction processes that satisfy the properties of MWPs to distill mathematical knowledge from LLMs to construct problem-equation pairs required for supervised training. In the Refinement Stage, we further utilize the unsuccessfully searched data effectively by a Knowledge Refine method. Finally, we train a small model using distilled data generated through two-stage methods. Our method demonstrates significantly improved performance on the Math23K and Weak12K datasets compared to existing small model methods, while maintaining a much lower computational cost than ChatGPT.
Introduction
Solving Math Word Problems (MWPs) is a cornerstone of Natural Language Processing that requires a model to bridge the gap between human language and formal mathematical logic. Traditionally, achieving high accuracy in this domain required "full supervision," where models are trained on datasets containing both the problem text and the step-by-step equations. However, creating such datasets is labor-intensive and expensive.
To mitigate these costs, researchers have turned to weakly supervised learning, where the model only receives the problem and the final numerical answer. While promising, this approach introduces the "false-matching problem": traditional search-based methods (like random walks) often find equations that happen to produce the correct answer but are logically unrelated to the problem's text. This paper introduces FLTT (From Large to Tiny), a framework designed to leverage the semantic reasoning of Large Language Models (LLMs) to train efficient, small-scale models that avoid these logical pitfalls.
The FLTT Framework
The FLTT framework operates on a "Large → Middle → Tiny" pipeline, utilizing a two-stage process to transfer expertise from a high-parameter teacher (ChatGPT) to a resource-efficient student model.
Stage 1: Knowledge Distillation
The first stage focuses on generating high-quality synthetic training data. By using Chain-of-Thought (CoT) prompting, the researchers prompt ChatGPT to generate equations for problems where only the answer is known. To ensure the student model learns mathematical structure rather than just memorizing constants, the framework employs a Masking process. Numbers in the problem and equations are replaced with placeholders (e.g., [Mask1], [Mask2]), forcing the model to focus on the underlying logic and operations.
Stage 2: Knowledge Refinement
LLMs are not infallible; they may produce incorrectly formatted outputs or fail to find a valid equation for certain problems. To maximize data utility, the authors introduce a "middle model" (T5-Base, 250M parameters). This model is first trained on the successful outputs from the distillation stage. It then iteratively attempts to solve the problems that the LLM initially failed. This refinement process creates a cleaner, more comprehensive dataset, effectively "smoothing" the knowledge for the final tiny model.
Experimental Results
The framework was evaluated on two primary benchmarks: Math23K (a standard Chinese MWP dataset) and Weak12K (a more challenging dataset designed specifically for weak supervision).
Performance vs. Efficiency
The results indicate that high-quality synthetic data can allow small models to outperform their much larger teachers:
- Surpassing the Teacher: On the Math23K dataset, the FLTT-T5 (95M) model achieved an accuracy of 71.8%, outperforming zero-shot ChatGPT, which scored 69.4%.
- State-of-the-Art in Weak Supervision: FLTT significantly outperformed previous weakly supervised methods. For instance, it surpassed ComSearch (60.5%) and WDA (55.1%) by substantial margins.
- Computational Gains: The final "tiny" models (as small as 15M parameters) maintain a fraction of the computational footprint of LLMs while delivering superior performance on these specialized tasks.
Discussion and Conclusion
The success of the FLTT framework highlights that semantic consistency is more critical than model size for MWP tasks. By using LLMs to ensure that equations actually match the problem's narrative, the framework resolves the false-matching issues inherent in previous search-based methods.
However, the researchers noted that data volume remains a factor; on the smaller Weak12K dataset, tiny models still lagged behind ChatGPT (46.1% vs 60.5%), suggesting that distillation requires a critical mass of data to fully bridge the gap. Future work will focus on automating the handling of LLM formatting inconsistencies and extending this distillation logic to more complex reasoning domains beyond basic mathematics. In conclusion, FLTT provides a viable path for deploying sophisticated mathematical reasoning in resource-constrained environments without the need for expensive manual annotations.
Summary updated