TL;DR
This paper introduces a novel method for evaluating the compositional generalization of Large Language Models (LLMs) in mathematics by combining existing problems into new ones. They found that even when the mathematical essence remains the same, LLMs struggle significantly with these combined problems, highlighting a key limitation in their reasoning abilities.
What Problem Does This Paper Address?
Evaluating the true mathematical reasoning capabilities of LLMs is a significant challenge. While LLMs have shown impressive performance on various math benchmarks, it's unclear if they've truly learned to reason compositionally or if they're relying on pattern matching or data leakage. Existing evaluation methods often create problem variants by perturbing a single problem. This paper addresses the gap by focusing on compositional generalization, specifically how LLMs handle problems that require combining knowledge from multiple, logically connected sub-problems. The authors argue that this ability is crucial for human-level intelligence and that current LLMs fall short when faced with such combinations, even if they can solve the individual components.
Methods
The core contribution is the Numerical-based Composition Synthesis Pipeline (NCSP). This pipeline automatically generates compositional math problems by:
- Handling Problem 1: Extracting a numerical value and its definition from the first problem's answer.
- Handling Problem 2: Masking a numerical value in the second problem and introducing an equivalent condition based on the first problem's extracted value.
- Establishing Relationships: Determining the relationship (e.g., greater than, equal to, less than) between the extracted values from Problem 1 and Problem 2.
- Renaming: Replacing variable names with more conventional ones to avoid artificiality.
The pipeline uses LLMs for generation and a Python interpreter (specifically, SymPy) for verification at each step, achieving a reported 98.2% accuracy in generating valid compositional problems.
Datasets
- VCMD (Verified Compositional Math Dataset): Synthesized using Llama-3.1-70b-it, consisting of 500 problems. Seed questions were drawn from the MATH dataset, with each seed question combined with others five times. Manually verified twice.
- ECMD (Extensive Compositional Math Dataset): Created by pairing seed questions that Llama-3.1-8b-it and Llama-3.1-70b-it answered correctly. Resulted in 30,818 valid problems.
- TCMD (Triplet Compositional Math Dataset): Formed by combining three correctly answered seed problems. Contains 6,878 problems.
Key Insights and Ideas
- Compositional Generalization is a Major Hurdle: Even when the mathematical essence of problems remains unchanged, simply combining them significantly degrades LLM performance. This suggests LLMs struggle to integrate knowledge from multiple sources in a novel way.
- Error Accumulation Amplifies Issues: The probability of errors in sub-problems multiplies in compositional tasks, leading to a much larger performance drop than expected.
- "Mastering" a Problem Isn't Enough: Humans can naturally combine problems they've mastered. LLMs, however, don't seem to exhibit this same intuitive generalization, even when sub-problem accuracy is high.
- Problem Characteristics Matter: Factors like high difficulty, low confidence in sub-problems, different problem types, long question length, and "easy-to-difficult" ordering pose greater challenges for compositional reasoning.
- RAWR (Right Answer, Wrong Reasoning) is a Concern: LLMs can sometimes arrive at the correct final answer through incorrect intermediate steps or flawed reasoning, highlighting the need for evaluating intermediate steps.
Key Results
- Experiments on 14 LLMs (ranging from 6B to 671B parameters) showed significant performance degradation on compositional problems compared to individual seed problems. For instance, some models experienced score drops of over 50%.
- The NCSP pipeline achieved 98.2% accuracy in generating valid compositional problems.
- Analysis of various factors revealed that difficulty, confidence, and length of sub-problems significantly impact compositional performance.
Limitations and Future Work
- Synthetic vs. Human-Created Problems: The compositional problems are synthesized, which may not represent the same level of novelty or challenge as human-created problems. However, they are deemed suitable for improving generalization.
- Pipeline Efficiency: The current pipeline requires multiple LLM calls for verification, which could be streamlined with more advanced models capable of multi-step reasoning in a single pass.
- Training Data Potential: The complex and difficult nature of these compositional problems makes them suitable for training reasoning models. The authors plan to implement this in future work.
- Further Investigation: The authors aim to further investigate LLMs in math to improve their compositional generalization capabilities.
Summary updated