TL;DR
This paper introduces MATHSENSEI, a novel tool-augmented large language model (LLM) designed to enhance mathematical reasoning capabilities. By strategically combining LLMs with external tools like web search, Python execution, and WolframAlpha, MATHSENSEI significantly outperforms existing LLMs on complex mathematical benchmarks, demonstrating the power of tool integration for advanced problem-solving.
What Problem Does This Paper Address?
Large Language Models (LLMs) have shown impressive capabilities in natural language processing, but they often struggle with complex reasoning tasks, particularly in mathematics. While techniques like Chain-of-Thought (CoT) prompting have improved performance, they fall short on problems requiring deep mathematical knowledge, intricate computations, or symbolic manipulation. This paper addresses the challenge of equipping LLMs with the necessary external tools and intelligent strategies to tackle these complex mathematical reasoning problems effectively. The authors aim to understand how different tool combinations and their sequencing impact performance across various mathematical domains and difficulty levels.
Methods
The core of this work is MATHSENSEI, a framework that augments LLMs with a suite of tools. These tools are categorized into:
- LLM-based modules:
- Knowledge Retriever (KR): Extracts relevant mathematical concepts, formulas, and hints from an LLM.
- Program Generator (PG): Generates Python code using the
sympylibrary to solve mathematical problems. - Code Refiner (CR): Iteratively corrects syntactical errors in the generated Python code.
- Solution Generator (SG): Compiles a step-by-step solution and provides the final answer.
- External APIs:
- Bing Web Search (BS): Retrieves relevant information, similar questions, and concepts from the web.
- WolframAlpha (WA): Leverages its computational engine for mathematical problem-solving.
MATHSENSEI explores various combinations and orderings of these modules. It also investigates planning strategies, including the non-adaptive "Plan-And-Solve" and the dynamic "REACT" (Reasoning and Acting) framework, to determine the optimal sequence of tool execution.
Datasets
The paper evaluates MATHSENSEI on several established mathematical reasoning datasets:
- MATH: A challenging dataset with 5000 problems across seven subjects (Precalculus, Prealgebra, Algebra, Geometry, Intermediate Algebra, Counting and Probability, Number Theory) and five difficulty levels.
- AQUA-RAT: Contains 253 algebraic math word problems with rationales, presented in a multiple-choice format.
- GSM-8K: A dataset of high school level math word problems requiring basic arithmetic operations.
- MMLU-Math: A subset of the MMLU benchmark focusing on mathematical questions across various levels, from elementary to college mathematics, also in a multiple-choice format.
Key Insights and Ideas
- Tool Synergy is Crucial: The effectiveness of MATHSENSEI stems from the complementary strengths of its tools. For instance, WolframAlpha excels at complex computations and symbolic manipulation, while Bing Search is useful for retrieving relevant context and similar problems.
- Complexity Matters: The benefit of using multiple tools is more pronounced for complex mathematical problems. Simpler datasets like GSM-8K show minimal improvement, whereas more intricate problems on MATH and MMLU-Math see substantial gains.
- Programmatic Solving is Powerful for Certain Domains: The Python Generator (PG) module, leveraging
sympy, significantly boosts performance on domains like Algebra and Intermediate Algebra, especially when dealing with complex computations and symbolic representations that natural language struggles with. - Web Search as a Knowledge Augmenter: Bing Web Search (BS) proves more effective than a purely LLM-based knowledge retriever (KR) for gathering relevant information, particularly for problems requiring external knowledge.
- Planning Strategies Need Refinement: While planning frameworks like Plan-And-Solve and REACT were explored, they did not consistently outperform well-configured static module sequences. This suggests that current planning methods may not be sufficiently adapted to the dynamic and unbounded nature of mathematical reasoning tasks.
Key Results
- MATHSENSEI achieves a significant improvement of 13.5% accuracy over
gpt-3.5-turbowith Chain-of-Thought prompting on the MATH dataset. - The best configuration, PG+WA+SG, achieved an accuracy of 47.6% on the MATH dataset.
- On Intermediate Algebra problems within MATH, the same configuration showed a +11.6% gain over GPT-4 with CoT prompting.
- For Precalculus problems on MATH, MATHSENSEI improved accuracy from 26.7% (GPT-4 with CoT) to 28.9%.
- The benefits of tool augmentation were less pronounced on simpler datasets like GSM-8K and AQUA-RAT, with improvements of only 2.4% and 3.3% respectively on MMLU-Math.
Limitations and Future Work
- Tool Set Limitations: The current toolset is focused on common mathematical disciplines. Future work could explore integrating more specialized solvers like Z3, SAT solvers, or commonsense knowledge bases.
- Program Generator Adaptability: The PG module's reliance on
sympymight be overkill for simpler problems. Future work could make the PG module adaptive, focusing on code generation only when necessary. - Planning Strategy Development: The authors acknowledge that current planning methods are not ideal for mathematical TALMs due to the unbounded action space and varying tool effectiveness. Developing more efficient and dynamic planning strategies tailored for mathematical reasoning is a key future direction.
- Bing Search Noise: The raw output from Bing Search API can be noisy, and a critique mechanism to assess the relevance of retrieved information is needed.
Summary updated