SAAS: Solving Ability Amplification Strategy for Enhanced Mathematical Reasoning in Large Language Models

Hyeonwoo Kim, Gyoungjin Gim, Yungi Kim, Jihoo Kim, Byungju Kim, Wonseok Lee, Chanjun ParkEMNLP 2024

TL;DR

Large Language Models (LLMs) struggle with mathematical reasoning. This paper introduces SAAS, a novel sequential learning strategy that first trains LLMs on Chain-of-Thought (CoT) to build logical skills, then fine-tunes them with Program-of-Thought (PoT) for problem-solving. SAAS achieves state-of-the-art performance, demonstrating the effectiveness of this structured learning approach.

What Problem Does This Paper Address?

While LLMs have made impressive strides in natural language processing, their mathematical reasoning capabilities lag significantly behind. This gap is a major hurdle for applications requiring logical thinking, problem-solving, and complex decision-making. Existing methods like Chain-of-Thought (CoT) improve reasoning but can lead to arithmetic errors with large numbers, while Program-of-Thought (PoT) addresses this by using code for computation but requires precise step-by-step coding. The challenge is to effectively combine these approaches to enhance both logical reasoning and computational accuracy in LLMs.

Methods

The core of this paper is the SAAS (Solving Ability Amplification Strategy), a sequential learning approach. SAAS operates in two main phases:

  1. Chain-of-Thought (CoT) Learning: The LLM is initially fine-tuned using CoT rationales. This phase focuses on building the model's ability to understand and generate logical reasoning steps, mirroring human problem-solving. The objective is to optimize model parameters for logically interpreting mathematical problems.
  2. Program-of-Thought (PoT) Learning with Cognitive Retention: Building on the logical foundation from CoT, the LLM is further fine-tuned using PoT rationales. To prevent "cognitive forgetting" (where the model loses previously learned reasoning skills), SAAS incorporates a cognitive retention strategy. This involves randomly sampling CoT rationales and mixing them into the PoT training data. This ensures the model retains its logical reasoning abilities while adapting to the algorithmic perspective of PoT.

The paper hypothesizes that this sequential order—CoT first, then PoT—is crucial for amplifying problem-solving abilities, as it establishes logical skills before tackling computational accuracy.

Datasets

The authors synthesized several datasets for training and evaluation, including:

  • GSM8K: Focuses on elementary math problems, testing logical reasoning and natural language understanding.
  • MATH: Covers a broad range of math problems from elementary arithmetic to calculus and geometry.
  • MetaMathQA: Augmented with rephrased questions, forward-backward reasoning, and self-verification.
  • MathInstruct: A mix of 13 types of CoT and PoT rationales across various mathematical fields.
  • QANDA: A diverse collection of real-world mathematical questions with detailed solutions.

They also mention using publicly available datasets like GSM-Hard, SVAMP, ASDIV, TabMWP, and MAWPS for out-of-domain evaluation.

Key Insights and Ideas

  • Sequential Learning is Key: The order of learning matters. Prioritizing logical reasoning (CoT) before computational accuracy (PoT) is more effective than simply combining them.
  • Cognitive Retention is Crucial: Simply switching to PoT learning can lead to a decline in mathematical reasoning abilities. The proposed cognitive retention strategy, which reintroduces CoT data, effectively mitigates this "forgetting."
  • CoT for Logic, PoT for Computation: CoT excels at breaking down problems logically but can falter on arithmetic. PoT excels at computation but requires precise coding. SAAS aims to leverage the strengths of both.
  • Domain-Specific Models Outperform General Models: For complex mathematical tasks, specialized models generally achieve better results than general-purpose LLMs of similar size.

Key Results

SAAS achieved state-of-the-art (SOTA) performance across several mathematical benchmark datasets.

  • Overall Performance: SAAS consistently outperformed competitors, often by significant margins (e.g., up to 5.26%, 7.71%, and 6.28% average score improvement on 7B, 7B-13B, and 13B-34B models respectively).
  • Effectiveness of Strategies:
    • Combining CoT and PoT learning significantly improved performance over individual CoT or PoT learning.
  • SAAS without cognitive retention showed only marginal improvement on some datasets, highlighting the importance of this strategy.
  • A "Reverse SAAS" (PoT then CoT) performed worse, underscoring the significance of the CoT-to-PoT learning order.
  • Difficulty Levels: SAAS showed improved performance across all categories of problem difficulty, particularly for problems requiring more reasoning steps.
  • Case Study: A qualitative analysis showed that CoT alone led to arithmetic errors, PoT alone could result in reasoning deficits, while SAAS demonstrated both accurate computation and enhanced mathematical reasoning.

Limitations and Future Work

  • Unpredictability of LLMs: LLMs can exhibit unpredictable behavior, which is challenging for precise mathematical reasoning.
  • Understanding Abstract Concepts: LLMs still struggle with abstract and complex mathematical reasoning, even if they perform well on structured problems.
  • Reliance on Synthetic Data: The use of synthetic data, while necessary, may not perfectly capture real-world scenarios, potentially leading to performance gaps.
  • Ethical Considerations: The potential misuse of AI and the need for responsible development remain ongoing concerns.

The authors suggest that continued research is needed to address these limitations and further advance LLMs' mathematical reasoning capabilities.

Summary updated

BibTeX
@inproceedings{kim2024saas,
  author = {Hyeonwoo Kim and Gyoungjin Gim and Yungi Kim and Jihoo Kim and Byungju Kim and Wonseok Lee and Chanjun Park},
  title = {{SAAS: Solving Ability Amplification Strategy for Enhanced Mathematical Reasoning in Large Language Models}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2024},
}
Figure