When evaluating a deep neural network, our natural instinct is to look at its final output—the text it generates, the class label it assigns, or the move it makes on a game board. However, treating a network as a pure black box obscures the rich internal computations occurring across its layers. As an input passes through a network, each layer transforms the data, building an increasingly sophisticated internal representation of the world.
To understand what a network "knows" at any given point during this forward pass, researchers use a technique called diagnostic probing (or representation probing). The core idea is to freeze the weights of the primary model, extract its internal hidden activations for a set of inputs, and train a lightweight, auxiliary classifier—a probe—to predict a specific semantic property or concept from those activations.
If the probe can successfully predict the concept, it demonstrates that the network has internally encoded that information. This approach allows us to look beyond task performance and directly audit the latent knowledge structures inside the model.
Why Linear Probes? The Geometry of Latent Space
A critical design choice in representation probing is the complexity of the probe itself. While a deep, non-linear neural network could extract almost any information from a high-dimensional vector space, doing so would blur the line between what the primary model knows and what the probe is learning to compute on its own.
For this reason, researchers predominantly use linear probes (such as logistic regression). A linear probe can only succeed if the target concept is already explicitly organized and linearly separable within the activation space. As Masaki Sakata et al. demonstrate in On Entity Identification in Language Models, language models naturally organize complex semantic concepts, such as entity clusters, into low-dimensional linear subspaces early in their processing layers. Similarly, Chenghao Xiao argues in Analyzing LLMs' Knowledge Boundary Cognition Across Languages Through the Lens of Internal Representations that a model's awareness of its own knowledge boundaries is encoded in a consistent, linear geometric structure across different languages. By restricting the probe to a simple linear transformation, we ensure that we are merely decoding information that the model has already formatted for its own internal use.
Mapping the Trajectory of Knowledge Across Depth
Neural networks process information hierarchically. By training independent probes on every individual layer, researchers can perform a layer-wise trajectory analysis, mapping exactly where and when specific concepts emerge, peak, or dissipate during a forward pass.
This layer-wise lens reveals a clear structural-to-semantic hierarchy across network depth:
- Linguistic Anomalies: Bai Li et al. showed in How is BERT surprised? Layerwise detection of linguistic anomalies that transformer layers act as an internal hierarchy for error detection. Surface-level grammatical errors are registered in early layers, while deeper semantic mismatches are caught much later.
- Concept Depth: Not all knowledge requires the same computational depth. Fan Yang et al. demonstrated in Exploring Concept Depth: How Large Language Models Acquire Knowledge and Concept at Different Layers? that simple factual associations are resolved in shallow layers, whereas complex reasoning and abstract emotional concepts require the deeper layers of the architecture to fully form.
- Context Integration: When a model encounters new information in a prompt that contradicts its pre-trained knowledge, layer-wise probing by Tianjie Ju et al. in How Large Language Models Encode Context Knowledge? A Layer-Wise Probing Study reveals how and where the model resolves this conflict across its hidden states to update its internal representations.
Disconnects Between Internal Knowledge and External Behavior
One of the most powerful insights gained from representation probing is that a model's internal representations often contain accurate information that fails to manifest in its final output. Probing exposes critical bottlenecks and misalignments between a model's latent knowledge and its behavior:
- Hidden Truths during Hallucination: Hadas Orgad et al. discovered in LLMs Know More Than They Show: On the Intrinsic Representation of LLM Hallucinations that even when a large language model generates a hallucinated or incorrect answer, a linear probe applied to its intermediate layers can often detect that the model internally possessed the correct fact. This reveals a failure of extraction rather than a lack of knowledge.
- Feature Prioritization: In computer vision, Katherine Hermann et al. used linear probing on frozen intermediate layers in The Origins and Prevalence of Texture Bias in Convolutional Neural Networks to show that models often encode structural shape information internally, even when their final classifications are heavily biased toward texture.
- The Vision-Language Handoff: When multimodal models fail to interpret charts or data visualizations, Alexa R. Tartaglini et al. used linear representation probing in Diagnosing Bottlenecks in Data Visualization Understanding by Vision-Language Models to show that the vision encoder successfully captures the precise data coordinates, but the downstream language module fails to retrieve or utilize this information correctly.
By revealing these hidden competencies, probing shows that a model's failure on a task does not necessarily mean it lacks the underlying concept; instead, the bottleneck may lie in how that information is routed to the final output layers.
Methodological Considerations and the Reliability Crisis
While representation probing provides an unprecedented window into neural networks, it is not without its challenges. Researchers must remain cautious about the stability of these internal signals. For instance, the authors of Do We Know What LLMs Don't Know? A Study of Consistency in Knowledge Probing highlight that probing classifiers can be highly sensitive to minor prompt perturbations or typos, sometimes yielding conflicting results.
Thus, while linear probing serves as an indispensable tool for mechanistic interpretability, ensuring the robustness and generalization of the probes themselves remains an active area of refinement.