Limitations of Language Models in Arithmetic and Symbolic Induction

Jing Qian, Hong Wang, Zekun Li, Shiyang Li, Xifeng Yan

Abstract

Recent work has shown that large pretrained Language Models (LMs) can not only perform remarkably well on a range of Natural Language Processing (NLP) tasks but also start improving on reasoning tasks such as arithmetic induction, symbolic manipulation, and commonsense reasoning with increasing size of models. However, it is still unclear what the underlying capabilities of these LMs are. Surprisingly, we find that these models have limitations on certain basic symbolic manipulation tasks such as copy, reverse, and addition. When the total number of symbols or repeating symbols increases, the model performance drops quickly. We investigate the potential causes behind this phenomenon and examine a set of possible methods, including explicit positional markers, fine-grained computation steps, and LMs with callable programs. Experimental results show that none of these techniques can solve the simplest addition induction problem completely. In the end, we introduce LMs with tutor, which demonstrates every single step of teaching. LMs with tutor is able to deliver 100% accuracy in situations of OOD and repeating symbols, shedding new insights on the boundary of large LMs in induction.

Introduction

Despite the "emergent" abilities of Large Language Models (LLMs), they often struggle with tasks that are computationally trivial for humans, such as adding multi-digit numbers or reversing long sequences. This research identifies two primary failure modes: Out-of-Distribution (OOD) Generalization and the Repeating Symbols problem. In OOD scenarios, models trained on short sequences (e.g., 5-digit addition) fail when tested on longer ones. The repeating symbols problem arises because the self-attention mechanism in Transformers struggles to distinguish between identical tokens at different positions, leading to a "locating" failure.

Methods

The study evaluates GPT-3 (via few-shot prompting) and T5 (via fine-tuning) across several mitigation strategies, including positional markers, fine-grained computation steps, and callable programs. However, the most successful approach is the novel LMs with Tutor method. This approach shifts the model's objective from predicting results to generating a sequence of actions—such as rmov (move cursor right) and cpy (copy)—mimicking the behavior of a multi-tape Turing machine. This abstracts the logic away from specific digits and focuses on algorithmic execution.

LMs with Tutor: Teaching models to generate actions like a Turing machine to solve symbolic tasks.

Key Results

The models were tested on synthetic datasets for copying, addition, and list reversal. While standard models perform well within their training distribution, their accuracy collapses in OOD settings. For instance, GPT-3's accuracy on addition drops to near zero as soon as the number of digits exceeds those provided in few-shot examples.

Performance of GPT-3 and T5 on addition and copy tasks, showing rapid degradation in out-of-distribution (OOD) scenarios.

Key findings include:

  • The Locating Bottleneck: The core issue is not a lack of mathematical knowledge but the inability of the attention mechanism to accurately track specific tokens in long or repetitive strings.
  • Tutor Method Success: Unlike other strategies, the LMs with Tutor approach achieved 100% accuracy across all tasks, regardless of sequence length or symbol repetition. By learning the rule of the algorithm, the model generalizes perfectly to OOD data.

Discussion and Future Work

While the "Tutor" method provides a robust solution for symbolic induction, it requires highly detailed training demonstrations, which may be difficult to scale to more complex, non-algorithmic reasoning tasks. Future work should explore how to integrate these algorithmic capabilities into general-purpose LLMs more efficiently, perhaps through architectural modifications that improve token resolution and tracking without requiring exhaustive step-by-step supervision.

Summary updated

Abstract

Recent work has shown that large pretrained Language Models (LMs) can not only perform remarkably well on a range of Natural Language Processing (NLP) tasks but also start improving on reasoning tasks such as arithmetic induction, symbolic manipulation, and commonsense reasoning with increasing size of models. However, it is still unclear what the underlying capabilities of these LMs are. Surprisingly, we find that these models have limitations on certain basic symbolic manipulation tasks such as copy, reverse, and addition. When the total number of symbols or repeating symbols increases, the model performance drops quickly. We investigate the potential causes behind this phenomenon and examine a set of possible methods, including explicit positional markers, fine-grained computation steps, and LMs with callable programs. Experimental results show that none of these techniques can solve the simplest addition induction problem completely. In the end, we introduce LMs with tutor, which demonstrates every single step of teaching. LMs with tutor is able to deliver 100% accuracy in situations of OOD and repeating symbols, shedding new insights on the boundary of large LMs in induction.
BibTeX
@inproceedings{qian2023limitations,
  author = {Jing Qian and Hong Wang and Zekun Li and Shiyang Li and Xifeng Yan},
  title = {{Limitations of Language Models in Arithmetic and Symbolic Induction}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
  url = {https://openreview.net/forum?id=sctKg1VD4S},
}
Figure