From Large to Tiny: Distilling and Refining Mathematical Expertise for Math Word Problems with Weakly Supervision

Qingwen Lin, Boyan Xu, Zhengting Huang, Ruichu Cai

Solving Math Word Problems (MWPs) is a classic challenge in Natural Language Processing. It requires a model to not only understand human language but also map that language to precise mathematical logic. Traditionally, this required "full supervision"—expensive datasets where every problem is paired with its step-by-step equation.

A new paper, "From Large to Tiny: Distilling and Refining Mathematical Expertise for Math Word Problems with Weakly Supervision," proposes a way to bypass this bottleneck. The researchers introduce FLTT (From Large to Tiny), a framework that uses the "brainpower" of Large Language Models (LLMs) like ChatGPT to train tiny, efficient models using only the final answers as a guide.

TL;DR

FLTT is a two-stage distillation framework that transfers mathematical reasoning from LLMs to "tiny" models (as small as 15M parameters). By using a "Distill and Refine" process, it creates high-quality synthetic training data that avoids the logical errors common in traditional search-based methods, eventually outperforming zero-shot ChatGPT on standard benchmarks.

What Problem Does This Paper Address?

Most MWP solvers are trained under weak supervision, meaning the model only sees the problem and the final answer, not the intermediate equation. To train, these systems usually use rule-based searches (like random walks) to find an equation that results in the correct answer.

The "Aha!" moment of the paper is identifying the false-matching problem: a rule-based search might find an equation that happens to produce the right number but is logically unrelated to the text. For example, if a problem asks for $10 + 2$, a search might find $6 \times 2$ and $10 + 2$ both equal 12, but only one is semantically correct. LLMs are great at avoiding this because they understand the context, but they are too computationally expensive to deploy in resource-constrained environments.

Methods: The FLTT Framework

The authors propose a "Large $\rightarrow$ Middle $\rightarrow$ Tiny" pipeline consisting of two main stages:

  1. Knowledge Distilling Stage: The researchers use ChatGPT to generate "problem-equation" pairs. To ensure the equations are useful for training smaller models, they use Chain-of-Thought (CoT) prompting and a Masking process. Instead of raw numbers, the equations use placeholders (e.g., [Mask1] + [Mask2]). This forces the model to learn the underlying mathematical structure rather than just memorizing specific numbers.

  2. Knowledge Refine Stage: LLMs aren't perfect; they sometimes fail to find a valid equation or produce weirdly formatted output. To capture the data ChatGPT missed, the authors train a "middle model" (T5-Base, 250M parameters) on the successful ChatGPT data. This middle model then iteratively searches for equations in the "failed" set. This creates a cleaner, more concise dataset for the final "tiny" model.

The Final Step: A tiny model (like RoFormer-Small at 15M parameters) is trained on this refined, distilled dataset.

Datasets

The authors evaluated their method on two major benchmarks:

  • Math23K: A standard dataset containing 23,161 Chinese math word problems.
  • Weak12K: A more difficult dataset of 12,117 problems specifically designed for weak supervision research. It is known for being a "stress test" where many models that perform well on Math23K usually struggle.

Key Insights and Ideas

  • Semantic Consistency is King: Unlike previous methods that just looked for "any equation that gets the right answer," FLTT leverages the LLM’s semantic understanding to ensure the equation actually matches the story in the problem.
  • The "Middle Model" Bridge: Using a 250M parameter model as a bridge between a 10B+ parameter LLM and a 15M parameter tiny model helps "smooth out" the knowledge, making it more digestible for the smallest architectures.
  • Iterative Success: The refinement stage showed that a model can "self-improve" by using its own successful predictions to go back and solve problems it previously found too difficult.

Key Results

The results demonstrate that you don't need a massive model to get high accuracy if your training data is high-quality:

  • Outperforming the Teacher: On the Math23K dataset, the FLTT-T5 (95M) model achieved an accuracy of 71.8%, actually beating zero-shot ChatGPT (69.4%).
  • SOTA in Weak Supervision: Compared to previous state-of-the-art weakly supervised methods like ComSearch (60.5%) and WDA (55.1%), FLTT showed a massive jump in performance.
  • Efficiency: The tiny models maintain a much lower computational cost than ChatGPT while providing comparable or superior mathematical reasoning on these specific tasks.

Limitations and Future Work

  • Data Volume Sensitivity: On the smaller Weak12K dataset, the tiny models still lagged behind ChatGPT (46.1% vs 60.5%). This suggests that while distillation is powerful, it requires a large volume of weakly supervised data to truly "surpass" the teacher.
  • Format Constraints: The authors noted that their "Format Correction" stage relies on identifying specific error types (like LaTeX notation or improper percentages). This manual enumeration can't catch every possible quirk in an LLM's output.
  • Future Direction: The authors suggest exploring more automated ways to handle format inconsistencies and applying this "Large to Tiny" logic to even more complex reasoning tasks beyond basic math.

Summary updated

Abstract

Addressing the challenge of high annotation costs in solving Math Word Problems (MWPs) through full supervision with intermediate equations, recent works have proposed weakly supervised task settings that rely solely on the final answer as a supervised signal. Existing leading approaches typically employ various search techniques to infer intermediate equations, but cannot ensure their semantic consistency with natural language descriptions. The rise of Large Language Models (LLMs) like ChatGPT has opened up new possibilities for addressing MWPs directly. However, the computational demands of LLMs make them less than ideal for use in settings where resources are tight. In light of these challenges, we introduce an innovative two-stage framework that adeptly transfers mathematical Expertise from large to tiny language models. In \emph{Distillation Stage}, we propose a series of extraction processes that satisfy the properties of MWPs to distill mathematical knowledge from LLMs to construct problem-equation pairs required for supervised training. In \emph{Refinement Stage}, Due to Knowledge distilling method cannot guarantee the full utilization of all data, we further utilize the unsuccessfully searched data effectively by Knowledge Refine method. Finally, We train a small model using distilled data generated through two-stage methods. As our method fully leverages the semantic understanding capabilities during the searching 'problem-equation' pair, it demonstrates significantly improved performance on the Math23K and Weak12K datasets compared to existing small model methods, while maintaining a much lower computational cost than ChatGPT.
BibTeX
@inproceedings{lin2024from,
  author = {Qingwen Lin and Boyan Xu and Zhengting Huang and Ruichu Cai},
  title = {{From Large to Tiny: Distilling and Refining Mathematical Expertise for Math Word Problems with Weakly Supervision}},
  booktitle = {ICIC (LNAI 6)},
  year = {2024},
  doi = {10.1007/978-981-97-5678-0\_22},
}

Loading related content...

Figure