Linear Probing of Internal Network Representations

1,182 concepts907 papers10 proposals

When evaluating a deep neural network, our natural instinct is to look at its final output—the text it generates, the class label it assigns, or the move it makes on a game board. However, treating a network as a pure black box obscures the rich internal computations occurring across its layers. As an input passes through a network, each layer transforms the data, building an increasingly sophisticated internal representation of the world.

To understand what a network "knows" at any given point during this forward pass, researchers use a technique called diagnostic probing (or representation probing). The core idea is to freeze the weights of the primary model, extract its internal hidden activations for a set of inputs, and train a lightweight, auxiliary classifier—a probe—to predict a specific semantic property or concept from those activations.

If the probe can successfully predict the concept, it demonstrates that the network has internally encoded that information. This approach allows us to look beyond task performance and directly audit the latent knowledge structures inside the model.

Why Linear Probes? The Geometry of Latent Space

A critical design choice in representation probing is the complexity of the probe itself. While a deep, non-linear neural network could extract almost any information from a high-dimensional vector space, doing so would blur the line between what the primary model knows and what the probe is learning to compute on its own.

For this reason, researchers predominantly use linear probes (such as logistic regression). A linear probe can only succeed if the target concept is already explicitly organized and linearly separable within the activation space. As Masaki Sakata et al. demonstrate in On Entity Identification in Language Models, language models naturally organize complex semantic concepts, such as entity clusters, into low-dimensional linear subspaces early in their processing layers. Similarly, Chenghao Xiao argues in Analyzing LLMs' Knowledge Boundary Cognition Across Languages Through the Lens of Internal Representations that a model's awareness of its own knowledge boundaries is encoded in a consistent, linear geometric structure across different languages. By restricting the probe to a simple linear transformation, we ensure that we are merely decoding information that the model has already formatted for its own internal use.

Mapping the Trajectory of Knowledge Across Depth

Neural networks process information hierarchically. By training independent probes on every individual layer, researchers can perform a layer-wise trajectory analysis, mapping exactly where and when specific concepts emerge, peak, or dissipate during a forward pass.

This layer-wise lens reveals a clear structural-to-semantic hierarchy across network depth:

Disconnects Between Internal Knowledge and External Behavior

One of the most powerful insights gained from representation probing is that a model's internal representations often contain accurate information that fails to manifest in its final output. Probing exposes critical bottlenecks and misalignments between a model's latent knowledge and its behavior:

By revealing these hidden competencies, probing shows that a model's failure on a task does not necessarily mean it lacks the underlying concept; instead, the bottleneck may lie in how that information is routed to the final output layers.

Methodological Considerations and the Reliability Crisis

While representation probing provides an unprecedented window into neural networks, it is not without its challenges. Researchers must remain cautious about the stability of these internal signals. For instance, the authors of Do We Know What LLMs Don't Know? A Study of Consistency in Knowledge Probing highlight that probing classifiers can be highly sensitive to minor prompt perturbations or typos, sometimes yielding conflicting results.

Thus, while linear probing serves as an indispensable tool for mechanistic interpretability, ensuring the robustness and generalization of the probes themselves remains an active area of refinement.

History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 839 papers
  2. Version 1 · 2026-10-04 · 707 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_representation_probing_bd04fc0b,
  title = {representation probing},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/representation-probing/rdr_bd04fc0b1a734f6e9a71b8607703e859},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 2, frozen text):

@misc{lacuna_representation_probing_bd04fc0b_v2,
  title = {representation probing},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/representation-probing/rdr_bd04fc0b1a734f6e9a71b8607703e859/v/2},
  note = {Version 2, frozen 2026-10-04. Accessed 2026-11-19},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake