How do autoregressive transformers solve full addition?

Wang Peixu, Chen Yu, Yu Ming, Cheng XiangEMNLP 2025

TL;DR

This paper investigates how autoregressive transformers, like those powering large language models (LLMs), perform addition. They discover that specific attention heads in middle layers are crucial for handling "carries" in addition, but struggle with long sequences. Interventions and fine-tuning can improve performance, revealing insights into how LLMs process arithmetic.

What Problem Does This Paper Address?

Large language models (LLMs) have shown remarkable capabilities across various tasks, but their internal workings remain largely a "black box." Understanding how they perform even basic arithmetic, like addition, is crucial for building trust, ensuring safety, and improving interpretability. While LLMs can often perform simple addition, they struggle with more complex cases, particularly those involving "carries" that propagate across multiple digits. This paper aims to demystify this process by analyzing the internal mechanisms of transformers when faced with addition tasks, especially those requiring carry operations.

Methods

The authors employ a multi-pronged approach:

  • Causal Analysis: They use causal analysis to pinpoint specific attention heads responsible for processing carry information. This involves manipulating attention weights to observe their impact on the model's output.
  • Inference Intervention: They directly intervene during inference by adjusting attention weights of identified heads. This is done without retraining the model to see if performance can be restored.
  • Fine-tuning: They fine-tune a smaller model (Gemma2-2B) specifically on addition tasks to observe how its internal mechanisms adapt.
  • Ablation Studies: They systematically remove or modify components (like specific attention heads) to understand their contribution to the addition task.
  • Analysis of Carry Chains: They differentiate between "Carry Chain" (CC) tasks (requiring carries) and "Only Chain" (OC) tasks (structurally similar but without actual carries) to isolate the challenges posed by carries.

Datasets

The paper uses existing pre-trained models and does not introduce new datasets. They evaluate on:

  • Synthetically generated addition problems: These include "Carry Chain" (CCd) and "Only Chain" (OCd) tasks, with varying lengths of carry chains (d).
  • Randomly generated number pairs: Used to ensure a broad evaluation of addition capabilities.

Key Insights and Ideas

  • Attention Heads for Carries: A small subset of attention heads, primarily in the middle layers, are identified as critical for handling the "carry" operation in addition. Different heads seem to specialize in carries of different lengths.
  • The "Staircase" Pattern: The model's attention mechanism for addition can be visualized as a "staircase" pattern, where information flows sequentially. This pattern breaks down for longer carry chains.
  • Local vs. Global Focus: The identified attention heads have a localized focus, which is effective for short carries but struggles to maintain global context for long-sequence additions.
  • Intervention Can Restore Performance: Targeted inference interventions on specific attention heads can significantly improve accuracy on longer addition tasks, demonstrating that the model can perform better with the right internal signals.
  • Fine-tuning Creates New Strategies: Fine-tuning on addition tasks leads to the emergence of specialized "Detection" and "Target" heads, suggesting the model can develop more efficient, task-specific mechanisms.
  • Length Generalization is a Major Hurdle: Despite improvements, models still struggle to generalize addition to lengths beyond their training maximum, indicating fundamental limitations in how they handle long-range dependencies.

Key Results

  • Mistral-7B: Inference intervention improved accuracy on 20-digit long-sequence additions from 2% to 38%.
  • Ablation Impact: Ablating the top two attention heads for CC1 tasks on Gemma-7B caused accuracy to drop from 99.3% to 1.47%, reverting the output to modular addition.
  • Performance Decline: As the length of the carry chain (d) increases, the model's ability to distinguish between CC and OC tasks diminishes significantly, with lines overlapping for d > 3.
  • Fine-tuning Gains: Fine-tuned Gemma2-2B showed improved accuracy on CCd tasks, though performance dropped sharply beyond the maximum training length.

Limitations and Future Work

  • Limited Scope: The study focuses solely on addition, while multiplication and division are more complex and require further investigation.
  • Tokenization Dependence: The proposed methods might need adjustments for models that don't use a "single-digit" tokenization approach (e.g., GPT-3.5/4).
  • Model Architecture Specificity: Conclusions are drawn from specific model architectures; training models from scratch with customized architectures could yield more generalizable insights.
  • Generalization Challenges: Achieving true length generalization for arithmetic tasks remains a significant challenge, likely due to limitations in positional encoding and the ability to form functional attention heads for arbitrary lengths.
  • Unclear Phase Change: The reason for the observed "phase change" in performance with increasing length remains unclear and warrants further study.

Summary updated

BibTeX
@inproceedings{peixu2025how,
  author = {Wang Peixu and Chen Yu and Yu Ming and Cheng Xiang},
  title = {{How do autoregressive transformers solve full addition?}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
}
Figure