Interpreting the Latent Structure of Operator Precedence in Language Models

Dharunish Yugeswardeenoo, Harshil Nukala, Ved Shah, Cole Blondin, Sean O Brien, Vasu Sharma, Kevin Zhu

TL;DR

This paper opens up the black box of how Large Language Models (LLMs) handle the mathematical order of operations. By analyzing LLaMA 3.2-3B, the authors demonstrate that models compute intermediate results internally within their hidden layers and linearly encode operator precedence immediately after the very first attention layer. They also introduce a novel "partial embedding swap" technique to successfully manipulate the model's internal mathematical logic.

What Problem Does This Paper Address?

While LLMs excel at language generation and coding, they notoriously struggle with basic arithmetic. Most existing research attempts to fix this from the outside using prompting strategies like Chain-of-Thought. However, the internal mechanistic structure of how these models process mathematical expressions remains poorly understood.

Specifically, do LLMs actually understand mathematical hierarchy (like PEMDAS), or do they just process equations in a linear, left-to-right sequence? This paper uses mechanistic interpretability to find out exactly where and how operator precedence is encoded inside an LLM's residual stream.

Methods

The authors employ a comprehensive suite of interpretability techniques on the open-source LLaMA 3.2-3B model:

  • Logit Lens: Projecting internal representations from intermediate layers directly into vocabulary space to see if intermediate answers (e.g., computing $3 \times 3 = 9$ within the expression $2 + 3 \times 3$) physically appear before the final answer is generated.
  • Linear Probing: Training simple linear models on top of layer activations to test if intermediate values are linearly decodable, even if they don't show up as the top token.
  • Attention Ablation: Systematically zeroing out attention outputs layer-by-layer to determine which layers are causally responsible for parsing structure versus performing the actual math.
  • Logistic Probing & UMAP: Visualizing and classifying operator tokens (like + or *) before and after attention blocks to see how their representations change based on their evaluation order.
  • Partial Embedding Swap (Novel): An intervention method that ranks embedding dimensions by their impact on precedence and swaps them between operators to see if they can force the model to switch from standard precedence to left-to-right evaluation.

Datasets

The authors constructed a synthetic dataset consisting of 8,547 arithmetic prompts containing three operands and two binary operators with mixed precedence (combinations of +, -, *, /). To isolate syntactic vs. semantic precedence, they generated six structural variations for each expression (e.g., left-parenthesized, right-parenthesized, no parentheses natural, no parentheses flipped).

To ensure they were analyzing successful mathematical reasoning pathways, they filtered the dataset down to 4,401 prompts where the model naturally predicted the correct final answer.

Key Insights and Ideas

  • MLPs do the math, Attention guides the structure: The logit lens revealed that intermediate results only become the top-ranked token after an MLP block, meaning MLP layers perform the actual computation. However, attention layers (especially layer 0) are critical for setting up the structural roadmap.
  • Precedence is established instantly: Right after the very first attention block (Layer 0), operator embeddings clearly separate based on their evaluation order. The model immediately tags which operator needs to be executed first.
  • Sparse and manipulable encoding: Operator precedence isn't distributed randomly across the entire embedding space; it is sparsely localized in specific dimensions. Swapping just a few key dimensions between a + and a * token can completely invert the model's execution order.

Key Results

  • Intermediate Tracking: In 63.6% of the correct cases, the intermediate calculation explicitly appeared as the top logit in the residual stream, peaking heavily around layers 18–19.
Linear probe R^2 and layer-wise detection frequency
  • Layer 0 Criticality: Ablating attention in Layer 0 caused the model's final answer accuracy to collapse, leading to incorrect outputs in 95.7% of cases, proving its foundational role in structural parsing.
Figure 2: Shows impact of attention ablation on output accuracy across 19 layers of the model. Early layers, particularly layer 0, showed the most pronounced effect on accuracy, suggesting a key role in establishing the structural interpretation of arithmetic expressions.
  • Perfect Linear Decodability: A logistic regression probe trained on post-attention Layer 0 activations predicted whether an operator would be evaluated first or second with 100% accuracy on the test set.

Limitations and Future Work

  • Model Scale: The study is limited to a relatively small model (LLaMA 3.2-3B); it remains to be seen if larger or frontier models utilize the same sparse geometric structures for math.
  • Expression Complexity: The dataset only includes simple, single-digit, three-operand expressions resulting in positive whole numbers. Real-world arithmetic involves negative numbers, decimals, and much longer sequences of operations.
  • Future Directions: The authors suggest that understanding these internal circuits could lead to methods for dynamically fixing arithmetic errors or investigating how models handle more complex algebraic structures.

Summary updated

Abstract

Large Language Models (LLMs) have demonstrated impressive reasoning capabilities but continue to struggle with arithmetic tasks. Prior works largely focus on outputs or prompting strategies, leaving the open question of the internal structure through which models do arithmetic computation. In this work, we investigate whether LLMs encode operator precedence in their internal representations via the open-source instruction-tuned LLaMA 3.2-3B model. We constructed a dataset of arithmetic expressions with three operands and two operators, varying the order and placement of parentheses. Using this dataset, we trace whether intermediate results appear in the residual stream of the instruction-tuned LLaMA 3.2-3B model. We apply interpretability techniques such as logit lens, linear classification probes, and UMAP geometric visualization. Our results show that intermediate computations are present in the residual stream, particularly after MLP blocks. We also find that the model linearly encodes precedence in each operator's embeddings post attention layer. We introduce partial embedding swap, a technique that modifies operator precedence by exchanging high-impact embedding dimensions between operators.
BibTeX
@inproceedings{yugeswardeenoo2025interpreting,
  author = {Dharunish Yugeswardeenoo and Harshil Nukala and Ved Shah and Cole Blondin and Sean O Brien and Vasu Sharma and Kevin Zhu},
  title = {{Interpreting the Latent Structure of Operator Precedence in Language Models}},
  booktitle = {INTERPLAY @ COLM},
  year = {2025},
  url = {https://openreview.net/forum?id=NFQ9C7lh6F},
}
Figure