TL;DR
This paper opens up the black box of how Large Language Models (LLMs) handle the mathematical order of operations. By analyzing LLaMA 3.2-3B, the authors demonstrate that models compute intermediate results internally within their hidden layers and linearly encode operator precedence immediately after the very first attention layer. They also introduce a novel "partial embedding swap" technique to successfully manipulate the model's internal mathematical logic.
What Problem Does This Paper Address?
While LLMs excel at language generation and coding, they notoriously struggle with basic arithmetic. Most existing research attempts to fix this from the outside using prompting strategies like Chain-of-Thought. However, the internal mechanistic structure of how these models process mathematical expressions remains poorly understood.
Specifically, do LLMs actually understand mathematical hierarchy (like PEMDAS), or do they just process equations in a linear, left-to-right sequence? This paper uses mechanistic interpretability to find out exactly where and how operator precedence is encoded inside an LLM's residual stream.
Methods
The authors employ a comprehensive suite of interpretability techniques on the open-source LLaMA 3.2-3B model:
- Logit Lens: Projecting internal representations from intermediate layers directly into vocabulary space to see if intermediate answers (e.g., computing $3 \times 3 = 9$ within the expression $2 + 3 \times 3$) physically appear before the final answer is generated.
- Linear Probing: Training simple linear models on top of layer activations to test if intermediate values are linearly decodable, even if they don't show up as the top token.
- Attention Ablation: Systematically zeroing out attention outputs layer-by-layer to determine which layers are causally responsible for parsing structure versus performing the actual math.
- Logistic Probing & UMAP: Visualizing and classifying operator tokens (like
+or*) before and after attention blocks to see how their representations change based on their evaluation order. - Partial Embedding Swap (Novel): An intervention method that ranks embedding dimensions by their impact on precedence and swaps them between operators to see if they can force the model to switch from standard precedence to left-to-right evaluation.
Datasets
The authors constructed a synthetic dataset consisting of 8,547 arithmetic prompts containing three operands and two binary operators with mixed precedence (combinations of +, -, *, /). To isolate syntactic vs. semantic precedence, they generated six structural variations for each expression (e.g., left-parenthesized, right-parenthesized, no parentheses natural, no parentheses flipped).
To ensure they were analyzing successful mathematical reasoning pathways, they filtered the dataset down to 4,401 prompts where the model naturally predicted the correct final answer.
Key Insights and Ideas
- MLPs do the math, Attention guides the structure: The logit lens revealed that intermediate results only become the top-ranked token after an MLP block, meaning MLP layers perform the actual computation. However, attention layers (especially layer 0) are critical for setting up the structural roadmap.
- Precedence is established instantly: Right after the very first attention block (Layer 0), operator embeddings clearly separate based on their evaluation order. The model immediately tags which operator needs to be executed first.
- Sparse and manipulable encoding: Operator precedence isn't distributed randomly across the entire embedding space; it is sparsely localized in specific dimensions. Swapping just a few key dimensions between a
+and a*token can completely invert the model's execution order.
Key Results
- Intermediate Tracking: In 63.6% of the correct cases, the intermediate calculation explicitly appeared as the top logit in the residual stream, peaking heavily around layers 18–19.
- Layer 0 Criticality: Ablating attention in Layer 0 caused the model's final answer accuracy to collapse, leading to incorrect outputs in 95.7% of cases, proving its foundational role in structural parsing.
- Perfect Linear Decodability: A logistic regression probe trained on post-attention Layer 0 activations predicted whether an operator would be evaluated first or second with 100% accuracy on the test set.
Limitations and Future Work
- Model Scale: The study is limited to a relatively small model (LLaMA 3.2-3B); it remains to be seen if larger or frontier models utilize the same sparse geometric structures for math.
- Expression Complexity: The dataset only includes simple, single-digit, three-operand expressions resulting in positive whole numbers. Real-world arithmetic involves negative numbers, decimals, and much longer sequences of operations.
- Future Directions: The authors suggest that understanding these internal circuits could lead to methods for dynamically fixing arithmetic errors or investigating how models handle more complex algebraic structures.
Summary updated