Abstract
We have recently witnessed a number of impressive results on hard mathematical reasoning problems with language models. At the same time, the robustness of these models has also been called into question; recent works have shown that models can rely on shallow patterns in the problem description when generating a solution. Building on the idea of behavioral testing, we propose a novel framework, which pins down the causal effect of various factors in the input, e.g., the surface form of the problem text, the operands, and math operators on the output solution. By grounding the behavioral analysis in a causal graph describing an intuitive reasoning process, we study the behavior of language models in terms of robustness and sensitivity to direct interventions in the input space. We apply our framework on a test bed of math word problems. Our analysis shows that robustness does not appear to continuously improve as a function of size, but the GPT-3 Davinci models (175B) achieve a dramatic improvement in both robustness and sensitivity compared to all other GPT variants.
Introduction
Large Language Models (LLMs) have demonstrated remarkable capabilities in solving complex mathematical word problems (MWPs). However, their underlying reasoning processes remain a subject of intense debate. Are these models truly "reasoning" through the logic of a problem, or are they simply exploiting shallow statistical correlations within the training data? Previous research has highlighted the "brittleness" of LLMs, where minor, logically irrelevant changes to a problem description—such as changing a character's name or slightly altering numerical values—can lead to catastrophic failures. This paper introduces a formal causal framework to disentangle true reasoning from pattern matching.
A Causal Framework for Reasoning
To rigorously evaluate model behavior, the authors propose a causal graph that models the intuitive reasoning process of a human solving a math problem. The framework identifies key variables: Operands ($N$), Operations ($O$), and the Surface Form ($S$) of the text.
By applying do-interventions (a concept from Judea Pearl’s Causal Calculus), the researchers manually manipulate these variables to see how the model's prediction ($R$) changes. They measure two specific effects:
- Total Causal Effect (TCE): Measures sensitivity, or how much the model's output changes when the ground-truth answer changes.
- Direct Causal Effect (DCE): Measures brittleness, or how much the model's output changes when input features (like numbers or text) change even though the correct answer remains the same.
An ideal reasoning agent should exhibit high sensitivity (high TCE) and high robustness (zero DCE).
Experimental Results and the "Davinci" Breakthrough
The framework was tested across 13 models, including GPT-Neo, GPT-J, and various GPT-3 iterations, using templates from benchmarks like ASDiv-A, MAWPS, and SVAMP. The results reveal a complex relationship between model size and reasoning capability.
As shown in the figure above, while increasing parameter count generally improves sensitivity (TCE), it does not automatically lead to better robustness. Most standard models remain highly brittle, showing significant DCE even as they grow larger. However, a distinct "phase change" is observed with the GPT-3 Davinci models (175B). These models, particularly the instruction-tuned variants like Davinci-003, show a dramatic leap in both sensitivity and robustness. This suggests that instruction tuning (e.g., RLHF) may be more critical for developing genuine mathematical reasoning than raw parameter count alone.
The Complexity Wall
Despite the success of Davinci-003, the study highlights a "complexity wall." When moving from simple two-operand problems to three-operand problems, even the most advanced models saw their brittleness (DCE) skyrocket from 0.17 to 0.87. This indicates that current LLMs still struggle to maintain logical consistency as the depth of reasoning increases. Furthermore, the study found that instruction-tuning LLaMA into "Alpaca" actually decreased its sensitivity while only slightly improving robustness, suggesting that the specific nature of the instruction-tuning data matters immensely.
Conclusion and Future Work
This causal approach provides a principled way to audit the reasoning capabilities of LLMs. The findings suggest that while scale helps, instruction tuning and architectural refinements are necessary to overcome the reliance on spurious correlations. Future research directions include applying this framework to "Chain-of-Thought" prompting to determine if step-by-step reasoning actually reduces the causal effect of irrelevant surface features and improves robustness in multi-step reasoning tasks.
Summary updated