Deep autoregressive language models are often criticized as inscrutable "black boxes." As an input token sequence passes through dozens of transformer layers, its representation is transformed into an abstract, high-dimensional vector space before a final linear layer—the unembedding matrix—projects it back into human-readable token probabilities.
But what if we didn't wait until the final layer to look at these probabilities?
The core intuition behind intermediate state decoding (often called the logit lens) is that a transformer's residual stream acts as an iterative refinement mechanism. Because the final step of a transformer is a linear projection into vocabulary space, we can apply that exact same linear transformation to the hidden states of intermediate layers. This allows us to inspect the model's "best guess" at any given depth, effectively translating abstract internal vectors into human-readable concepts mid-computation.
Why It Works: The Intuition of the Residual Stream
To understand why this works, consider how a transformer is structured. Instead of completely replacing the representation at each layer, transformer blocks write their outputs additively to a shared highway called the residual stream. By the time the vector reaches the end of the network, a linear unembedding matrix $W_U$ maps this accumulated representation directly to a vector of logits over the vocabulary.
Because matrix multiplication is distributive over addition, the final logit distribution is simply the sum of the projections of each individual layer's contribution. Therefore, applying $W_U$ directly to an intermediate hidden state $h_\ell$ yields a meaningful snapshot of what that layer has contributed to the final prediction. As Mor Geva et al. demonstrate in Transformer Feed-Forward Layers Build Predictions by Promoting Concepts in the Vocabulary Space, individual feed-forward networks (FFNs) act as "concept boosters," systematically shifting vocabulary probabilities toward specific semantic clusters long before the final layer.
Tracking the Layer-Wise Evolution of Knowledge
By projecting intermediate states across every layer, researchers can trace the exact pipeline the model uses to solve complex tasks. For example, when retrieving factual associations (e.g., querying "The capital of France is..."), Mor Geva et al. show in Dissecting Recall of Factual Associations in Auto-Regressive Language Models that early layers enrich the subject token with related attributes, while middle-to-late layers actively extract and promote the target relation.
This progressive refinement is visible across diverse domains:
- Mathematics and Logic: In How does GPT-2 compute greater-than?: Interpreting mathematical abilities in a pre-trained language model, Michael Hanna et al. use intermediate projections to visualize how the probability distribution of potential numerical outputs is narrowed down layer by layer. Similarly, Kevin Zhu demonstrates in Interpreting the Latent Structure of Operator Precedence in Language Models that models transition linearly from structural parsing to active computation within the residual stream.
- Reasoning Chains: When executing multi-step reasoning, models maintain latent representations of partial computations. Chengzhi Li et al. in Memory or Reasoning? Explore How LLMs Compute Mixed Arithmetic Expressions show that intermediate results of sub-expressions are physically represented and decodable at specific middle layers before the final answer is computed.
Exposing Latent States and Internal Pivots
One of the most powerful aspects of vocabulary projection is its ability to reveal "hidden" computations that never make it to the final text output. This exposes fascinating properties of how models generalize and regulate themselves:
- The English Pivot: Multilingual models often perform internal reasoning in an English-centric concept space. Chris Wendler et al. in Do Llamas Work in English? On the Latent Language of Multilingual Transformers and Yarin Gal et al. in Do Multilingual LLMs Think In English? decoded intermediate layers to find that non-English inputs are frequently translated into English internally before being mapped back to the target language in the final layers.
- Safety and Suppression: Alignment mechanisms often act as late-stage filters rather than erasing underlying capabilities. Asma Ghandeharioun et al. show in Who's asking? User personas and the mechanics of latent misalignment that harmful or suppressed responses are often fully formed in intermediate layers before being overridden by late-stage refusal tokens. This pattern is mirrored in safety alignment studies by Haiyang Yu et al. in How Alignment and Jailbreak Work: Explain LLM Safety through Intermediate Hidden States.
From Interpretation to Intervention
Understanding how predictions mature across layers has allowed researchers to move from passive observation to active performance enhancement.
If higher-level semantic knowledge is concentrated in deeper layers while early layers contain superficial linguistic patterns, we can contrast these distributions to improve decoding. For instance, Chufan Shi et al. in A Thorough Examination of Decoding Methods in the Era of LLMs explore cross-layer logit contrasting, which amplifies mature factual knowledge by subtracting intermediate layer logits from final layer logits during generation.
Furthermore, identifying where correct information is lost helps mitigate hallucinations. In Reefknot: A Comprehensive Benchmark for Relation Hallucination Evaluation, Analysis and Mitigation in Multimodal Large Language Models, Kening Zheng et al. use logit lens diagnostics to pinpoint where relational hallucinations originate, allowing them to steer the model back toward grounded representations preserved in earlier layers.