A Causal Framework to Quantify the Robustness of Mathematical Reasoning with Language Models

Alessandro Stolfo, Zhijing Jin, Kumar Shridhar, Bernhard Schölkopf, Mrinmaya Sachan

Abstract

Alessandro Stolfo, Zhijing Jin, Kumar Shridhar, Bernhard Schoelkopf, Mrinmaya Sachan. Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

2023.

Introduction

Large Language Models (LLMs) have demonstrated remarkable performance on mathematical benchmarks. However, a persistent concern remains: are these models genuinely reasoning through the logic of a problem, or are they merely exploiting statistical regularities in their training data? This paper, presented at ACL 2023, introduces a formal causal framework to move beyond simple accuracy scores. By applying causal inference, the authors quantify the extent to which LLMs rely on mathematical logic versus "spurious correlations" like problem phrasing.

Methodology: Causal Graphing and Interventions

The researchers define a causal graph to distinguish between the intended reasoning path and shortcut paths that models might take. In a "Human Reasoning" path, the question leads to specific operands and operations, which determine the result. Conversely, a model might bypass logic by mapping the "surface form" of a question directly to a prediction.

Causal graph illustrating the intended reasoning path (blue) versus potential spurious correlations (red).

To disentangle these effects, the authors employ Pearl’s do-calculus to perform controlled interventions on problem inputs:

  • Total Causal Effect (TCE): This measures sensitivity. By changing the operands ($N$) such that the ground-truth answer ($g$) changes, the authors observe if the model correctly updates its prediction.
  • Direct Causal Effect (DCE): This measures brittleness. By changing the operands or the text ($S$) while keeping the ground-truth answer the same, they test if the model is distracted by irrelevant changes.

The study utilizes templates from ASDiv-A, MAWPS, and SVAMP to generate thousands of intervention pairs, focusing on arithmetic problems with two or three operands.

Key Results: The Scaling Threshold

The most significant finding is that mathematical robustness is not a linear function of model size. Smaller models, including GPT-2 and GPT-Neo variants up to 20B parameters, remain highly brittle, showing high sensitivity to irrelevant changes.

The gap between sensitivity (TCE) and brittleness (DCE) across various model scales. Note the dramatic improvement in GPT-3 Davinci-003.

As shown in the results, the 175B parameter GPT-3 models (specifically Davinci-003) exhibit a non-linear leap in performance. While earlier models show a narrow gap between TCE and DCE, Davinci-003 achieves a wide separation, indicating it is far more likely to change its answer due to mathematical logic than superficial phrasing.

Discussion and Insights

  • Instruction Tuning: While instruction tuning generally improves robustness, the nature of the tuning data is critical. For instance, Stanford's Alpaca model showed decreased mathematical sensitivity compared to its base LLaMA model, likely due to a lack of reasoning-intensive data in its tuning set.
  • Complexity Wall: Even the most advanced models struggle with increased complexity. Moving from two-operand to three-operand problems results in a significant drop in robustness, suggesting that current LLM reasoning is still fragile.

Conclusion

This work provides a principled method for evaluating the "reasoning" in LLMs. The results suggest that while smaller models may act as "stochastic parrots," true mathematical reasoning capabilities appear to be an emergent property at massive scales, further enhanced by specific instruction tuning. Future work will extend this framework to multi-step reasoning and more complex mathematical domains.

Summary updated

BibTeX
@inproceedings{stolfo2023causal,
  author = {Alessandro Stolfo and Zhijing Jin and Kumar Shridhar and Bernhard Schölkopf and Mrinmaya Sachan},
  title = {{A Causal Framework to Quantify the Robustness of Mathematical Reasoning with Language Models}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
  url = {https://openreview.net/forum?id=j4XstJpJps},
}
Figure