Abstract
Large Language Models (LLMs) are increasingly used for reasoning tasks, yet their performance in basic arithmetic remains inconsistent. This paper investigates two primary factors influencing arithmetic accuracy: the language of the prompt and the efficacy of multi-agent verification strategies. By testing models like GPT-4, PaLM-2, and Gemini across 20 languages and numerical ranges up to 16 digits, we demonstrate that arithmetic is not language-neutral for LLMs. Furthermore, we show that while cross-agent checking can assist weaker models, it rarely exceeds the performance of a single high-capacity model.
Introduction
The assumption that mathematics serves as a "universal language" suggests that the linguistic framing of a problem should not affect its solution. However, LLMs operate on token prediction rather than algorithmic calculation. This study explores whether the linguistic context—specifically the language used in a prompt—impacts a model's ability to perform addition and subtraction. Additionally, we evaluate the "self-correction" paradigm, testing whether multi-agent collaboration or self-verification can mitigate the inherent errors found in LLM arithmetic.
Methodology
The researchers developed a comprehensive evaluation framework involving a new dataset of 39,708 instances, split equally between addition and subtraction. The methodology focused on:
- Multilingual Prompting: A standardized arithmetic prompt was translated into 20 languages, including English, Spanish, Russian, Japanese, Hindi, and Thai.
- Varying Complexity: The test set included numbers ranging from 1 to 16 digits to identify performance thresholds.
- Verification Strategies: Three distinct setups were compared: Zero-shot (direct answering), Self-Checking (internal verification), and Cross-Agent Checking (one model answering, another verifying).
- Model Selection: Evaluations were conducted on high-performing models including GPT-3.5, GPT-4, PaLM-2, and Gemini.
The Linguistic Influence on Arithmetic
The study reveals a significant "Language Gap" in arithmetic performance. For GPT-3.5, there was an 11.22% accuracy difference between English (62.44%) and Hindi (51.32%). Interestingly, linguistic proficiency in arithmetic does not always mirror general language performance. While English often leads, certain languages like Thai showed superior performance in specific operations like addition. For more advanced models like GPT-4, Russian and Japanese actually outperformed English, suggesting that the "English-first" bias in training data is not a universal constant for all reasoning tasks.
As shown in the data, the choice of language can shift a model's rank and accuracy significantly, suggesting that the training data distribution and tokenization of numbers in different scripts play a crucial role in numerical reasoning.
Complexity and the "Borrowing" Hurdle
Performance is not only language-dependent but also highly sensitive to the nature of the operation. Models consistently struggle more with subtraction than addition, often due to the "borrow" operation. While addition requires "carrying," which LLMs handle relatively well for small numbers, the logic of "borrowing" in subtraction presents a major hurdle, leading to a ~20% accuracy gap in some cases.
Furthermore, there is a clear "Digit Wall." Accuracy remains high for numbers up to five digits but drops precipitously as the number of digits increases. For 16-digit addition, GPT-3.5's accuracy falls to roughly 33.61%. This suggests that LLMs rely heavily on memorized patterns for common, small-scale arithmetic but lack a robust, scalable internal algorithm for multi-digit computation.
The Limits of Multi-Agent Collaboration
A popular hypothesis in LLM research is that multi-agent systems can "self-correct" errors. However, this study finds that collaboration is not a silver bullet for arithmetic.
While cross-agent checking (e.g., using PaLM-2 to verify GPT-3.5) improved the weaker model's performance by 10.81%, the resulting accuracy (73.25%) still fell short of PaLM-2's solo performance (81.51%). In many instances, the collaborative approach introduced new errors or failed to catch existing ones, indicating that a single strong model is often more reliable than a committee of weaker ones.
Conclusion
Arithmetic proficiency in LLMs is deeply tied to linguistic context and numerical complexity. The significant variance across languages challenges the view of math as a universal task for AI. Moreover, the limited success of cross-agent checking underscores the need for better fundamental reasoning capabilities or the integration of external tools like calculators. Future work should focus on developing language-agnostic training methods to ensure consistent numerical reasoning across all linguistic backgrounds.
Summary updated