TL;DR
Researchers have developed a way to pack the mathematical reasoning of giant models like GPT-4 into "Small Language Models" (SLMs) with fewer than a billion parameters. By teaching these tiny models to express problems as systems of equations—a method called Equation-of-Thought (EoT)—and combining them with code and text reasoning, they achieved state-of-the-art performance, sometimes even outperforming models ten times their size.
What Problem Does This Paper Address?
Large Language Models (LLMs) are brilliant at math, but they are computationally "heavy." Running a model like GPT-3 requires hundreds of gigabytes of memory and multiple high-end GPUs, making them inaccessible for local deployment or edge devices.
Small Language Models (SLMs) are the solution for democratization, but they usually fail at math for two reasons:
- Calculation Errors: They understand the logic but mess up the arithmetic (e.g., $7 + 7 = 12$).
- Logic Gaps: They struggle to maintain a coherent "Chain-of-Thought" (CoT) over multiple steps.
While previous methods like Program-of-Thought (PoT)—which asks models to write Python code—helped, they often failed if the model forgot to define a single variable. This paper seeks a more robust way to distill complex reasoning into these compact architectures.
Methods
The authors introduce two primary frameworks:
- Equation-of-Thought Distillation (EoTD): Instead of writing a paragraph or a full Python script, the model is trained to generate a system of linear equations. This is a "middle ground" that is more structured than text but less rigid than code. Crucially, the model doesn't solve the equations itself; it passes them to an external deterministic equation solver. This offloads the "cognitive load" of arithmetic from the model to a tool that never makes a math error.
- Ensemble Thoughts Distillation (ETD): The authors realized that different problems suit different reasoning styles. They created a diverse dataset containing three types of reasoning:
- CoT: Natural language step-by-step.
- PoT: Python programs.
- EoT: Systems of equations.
- Hierarchical Inference: At test time, the model uses a "fallback" strategy. It first tries to generate a Program (PoT). If the code doesn't compile, it tries Equations (EoT). If the solver can't handle the equations, it falls back to a standard text rationale (CoT).
Datasets
The researchers used the GSM8K training set as the foundation, using ChatGPT (gpt-3.5-turbo) to generate the various reasoning paths (CoT, PoT, EoT). After rigorous filtering (dropping any generated path that led to a wrong answer), they were left with a high-quality distillation dataset.
For evaluation, they tested on four major benchmarks:
- GSM8K: Grade school math word problems.
- ASDiv: Diverse math word problems.
- SVAMP: Challenging variations of math problems.
- MultiArith: Multi-step arithmetic problems.
Key Insights and Ideas
- Delegation is Efficiency: The most successful SLMs aren't the ones that are "smarter" at math, but the ones that are better at using tools. By training the model to be a "translator" (from English to Equations) rather than a "calculator," performance skyrockets.
- Diversity Prevents Fragmentation: Earlier distillation methods focused on a single format. The authors found that training on an ensemble (ETD) provides a "richer knowledge of reasoning," allowing the model to understand mathematical relationships from multiple perspectives.
- The "Unknown Variable" Advantage: PoT (code) often fails because Python requires every variable to be assigned a value before use. EoT (equations) allows the model to define a variable as $x$ and solve for it later, which is much closer to how humans naturally solve algebra.
Key Results
The results for the CodeT5 family of models (ranging from 60M to 770M parameters) were striking:
- Punching Above Their Weight: The CodeT5-Large (770M) model using the ETD method achieved 42.45% accuracy on GSM8K. For context, this is significantly higher than much larger 7-billion parameter models like Llama-2 (13.3%) and Platypus-2 (14.4%).
- Massive Gains: For the smallest model (60M parameters), adding EoTD moved the needle from a near-zero 1.1% accuracy to 18.87%. Adding the full ETD ensemble pushed it further to 33.58%.
- Robustness: The hierarchical approach (PoT → EoT → CoT) ensured that even if the model's first logic path failed, it had multiple "safety nets" to find the correct answer.
Limitations and Future Work
- Pre-training Bias: Most SLMs are pre-trained on massive amounts of natural language and code, but very little "equation-system" data. This makes it harder for the models to learn EoT compared to CoT or PoT.
- Data Generation Difficulty: When generating the training data from LLMs, the "drop rate" for EoT was 46.7% (meaning nearly half of the LLM's attempts to create equations were wrong or unsolvable), suggesting that even giant models find equation-mapping difficult.
- Future Directions: The authors plan to explore if this "Equation-of-Thought" approach can be applied to fields beyond mathematics, such as formal logic or scientific reasoning.
Summary updated