For years, the standard way to understand what a "black box" model like BERT knows was to train a probe. If you wanted to see if BERT understood parts of speech, you would freeze its weights, take its internal vectors, and train a small classifier (the probe) to predict nouns and verbs. If the probe succeeded, we concluded that BERT "knew" grammar.
However, a growing body of research suggests this logic is flawed. As Dan Friedman and Koustuv Sinha have argued, a sufficiently powerful probe can learn to solve a task using only the statistical regularities of the data, even if the underlying model is completely random. This creates a "probing paradox": is the knowledge inside the model, or is the probe just a very good student?
To solve this, researchers are moving toward intrinsic, parameter-free methods. Instead of training new layers to "read" the model, they are looking at the model's own internal geometry, mathematical properties, and emergent behaviors to see what it truly understands.
The Problem of Probe Capacity
The fundamental critique of supervised probing is that it conflates representation with extractability. Joelle Pineau and Sarath Chandar point out that non-linear probes can "hallucinate" knowledge by learning the task themselves. If a probe has enough parameters, it can map even noisy, unrelated embeddings to the correct labels.
This risk is particularly high in "factual probing." Dan Friedman demonstrated that optimized probes often learn to exploit patterns in the training set rather than recalling facts from the language model. To combat this, researchers like Victor Sanh and Alexander M Rush proposed subnetwork probing. Instead of adding new parameters, they look for a "winning ticket"—a subset of the model's existing weights—that can perform the task. If the knowledge is already there, you shouldn't need to add anything new to find it.
Measuring Geometry Without Classifiers
If we can't trust trained classifiers, how do we measure what's inside? One answer lies in the geometry of the embedding space.
Vivek Srikumar and Yichu Zhou introduced DirectProbe, which bypasses classifiers entirely. It examines how clusters of data points (like "verbs" vs "nouns") are physically arranged in the high-dimensional space. By looking at the distance and "convex hulls" of these clusters, they can determine how well-separated and organized the model's internal categories are without training a single extra parameter.
Similarly, Shikhar Murty and Jacob Andreas developed Tree Projections. Instead of asking a probe to "find a tree," they measure how much the model's internal computations naturally follow a tree-like, hierarchical structure. They found that as models train, their internal logic becomes more "tree-like" on its own, which is a better predictor of generalization than simple accuracy.
Zero-Shot and Behavioral Probing
Another way to avoid the "smart probe" trap is to use tasks that require no training at all. Maureen de Seyssel and her colleagues at Apple adapted the ABX discrimination task from speech science. In this framework, you simply ask: "Is representation A more similar to B or X?" By measuring these raw similarity distances, they can track how a model like XLM-R separates language "form" from "meaning" across its layers without any supervised labels.
Najoung Kim and Alex Warstadt took a different approach with Reconstruction Probing. Instead of external labels, they use the model's own objective: how much does the representation of one word help the model guess the words around it? This "behavioral" probe revealed that most contextual information is actually present in the very first layer, challenging the idea that context is "built up" slowly through deep layers.
Spectral Signatures and Latent Topologies
The most recent frontier involves treating the model as a mathematical object rather than a black box. Chris Russell introduced CAST, which analyzes the "spectral signatures" of the matrices that map one layer to the next. By looking at the eigenvalues of these transformations, they can see whether a layer is compressing information or expanding it, providing a "probe-free" map of the model's functional architecture.
For complex structures like grammar, Mirella Lapata and Yao Fu proposed Latent Topology Induction. They treat the model's hidden states as nodes in a graph. By analyzing the transitions between these states as the model processes a sentence, they can induce the "latent grammar" the model is using. This reveals that models aren't just matching patterns; they are traversing a structured internal map of language.
Why It Works: The Shift to Inherent Logic
The shift from supervised probing to intrinsic analysis works because it removes the "middleman." When we use PARAFAC2 joint matrix factorization (as explored by Zheng Zhao and Shay B Cohen) or SVCCA (used by Naomi Saphra), we are measuring the alignment and correlation of the model's own internal signals.
By prioritizing the model's raw geometry over a probe's performance, researchers can finally distinguish between:
- Stochastic Parrots: Models that store a syntactic history that a probe can exploit.
- World Models: Models that develop internal, emergent representations of formal semantics, such as the program execution states found by Charles Jin and Martin Rinard.
The goal of this research direction is to ensure that when we say a model "understands," we are describing the model's own internal reality, not just the cleverness of our diagnostic tools.