Probing the Internal Representations of Large Language Models

948 concepts812 papers46 proposals

Large Language Models (LLMs) have achieved remarkable success across highly complex domains, including finance (Ding et al.), healthcare (He et al.), and telecommunications (Piovesan et al.). Yet, their internal decision-making processes remain notoriously opaque. This "black box" problem introduces severe engineering risks, such as unpredictable software integration failures (Cai et al.; Shao et al.) and vulnerability to adversarial manipulation (Brambilla et al.).

To move beyond post-hoc behavioral observations, a growing body of research focuses on probing and mechanistic interpretability. Instead of merely analyzing what an LLM outputs, these techniques look directly at the hidden states, layer dynamics, and individual neurons to understand why and how the model arrives at its conclusions.

Probing the Geometry of Hidden States

How do we determine if an LLM truly understands a concept or is simply executing surface-level pattern matching? One powerful approach is linear probing—training a simple linear classifier on the model's internal activations to see if specific information is explicitly represented.

For example, Zhu et al. demonstrated that LLMs encode the magnitude of numbers linearly within their hidden states. By extracting these representations, the authors showed a direct causal link between this linear encoding and the model's ability to perform arithmetic computations. Similarly, structural probes can reveal abstract linguistic hierarchies. Mysiak et al. utilized structural probes to analyze dependency tree structures in multilingual models, proving that the models implicitly map language family groups without explicit supervision.

Beyond individual concepts, the global geometry of how an LLM processes information follows a highly consistent mathematical trajectory. By analyzing 28 different open-weight models, Joshi et al. discovered a universal "hunchback" pattern in the intrinsic dimensionality of hidden states. As data flows through the transformer layers, the dimensionality first expands to process the semantic variety of the input, and then sharply compresses as the model "commits" to a specific token decision.

Localizing Circuits and Cornerstone Layers

If the hidden states contain structured geometry, how do the individual layers and neurons collaborate to build it? Mechanistic interpretability treats the transformer architecture as an auditable circuit, localizing specific tasks to distinct sub-components.

Not all layers contribute equally to a model's intelligence. Zhang et al. computed layerwise Shapley values and discovered "cornerstone layers"—critical early-stage layers that act as computational bottlenecks. While many intermediate layers exhibit high redundancy and can be pruned safely, removing a single cornerstone layer causes the entire model to collapse into random guessing.

For higher-level cognitive tasks, researchers use techniques like path patching to isolate reasoning pathways. The researcher Abhinav Joshi applied this to map the exact transformer layers responsible for commonsense procedural reasoning, converting complex human scripts into directed acyclic graphs to trace the internal decision logic.

This localization can even reach the single-neuron level. In Retrieval-Augmented Generation (RAG) systems, a model must constantly choose between its pre-trained parametric memory and external retrieved context. Wang et al. identified a four-stage "knowledge stream" across layers and isolated the specific neurons that control this prioritization. Deactivating these neurons allows developers to explicitly dictate whether the model trusts its own memory or the provided text.

Validating Internal Mechanics Against External Behavior

Understanding internal mechanics provides a rigorous foundation for explaining macro-level behavioral phenomena. For instance, psychological evaluations show that LLMs mirror human cognitive biases, such as the "mental number line" (Shaki et al.), and display distinct ideological leanings when faced with ambiguous queries (Manari Hirose). Probing bridges the gap between these behavioral observations and structural realities.

Furthermore, monitoring internal states enables advanced quality assurance. Ouyang et al. introduced a "Surprise Adequacy" metric that tracks internal neuron activation patterns to detect subtle adversarial inputs that easily bypass standard black-box behavioral checks. By shifting the focus from what the model says to how its internal circuits fire, interpretability research turns unpredictable black boxes into controllable, verifiable software components.

History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_llm_interpretability_275a92f3,
  title = {LLM interpretability},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/llm-interpretability/rdr_275a92f3679f4badbe3bde4f15f2fb5f},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_llm_interpretability_275a92f3_v1,
  title = {LLM interpretability},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/llm-interpretability/rdr_275a92f3679f4badbe3bde4f15f2fb5f/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake