TL;DR
The paper introduces calibrated concept profiles, a framework that transforms direct neuron reading from a subjective inspection into a rigorous, unit-based measurement. By using controlled stimulus distributions and standardized units, it allows researchers to track how specific concepts (like Python for loops) emerge and evolve across a model's entire training history without needing to train secondary models like sparse autoencoders.
What Problem Does This Paper Address?
In the quest to understand the "black box" of Large Language Models (LLMs), the field of interpretability has largely moved away from looking at individual neurons. Direct reading was plagued by "polysemanticity" (neurons having multiple meanings), a lack of standardized units, and results that didn't transfer between studies. Consequently, researchers turned to "second models"—like linear probes or sparse autoencoders (SAEs)—to interpret the first.
However, this move comes at a cost: you lose the ability to name the model's native units directly and perform simple causal tests (like ablating a specific neuron). This paper argues that direct neuron reading isn't fundamentally flawed; it just lacks the rigor of a formal measurement. The authors aim to bring that rigor back by defining clear populations of stimuli, standardized units, and validation "licences."
Methods
The authors propose a six-step process to create a calibrated concept profile for any given construct:
- Prompt Generator: Instead of hand-writing a few examples, they generate a "battery" of prompts based on a declared distribution measured from a real-world reference corpus. They create three sets: Set A (the target concept in its role), Set B (the same token but in a control role, like inside a comment), and Set C (baseline ordinary code).
- Raw Readings: They calculate the difference in mean activation between Set A and Set B. This isolates the neuron's response to the role of the concept, effectively subtracting the "noise" of the token itself.
- The Profile (Standardization): To compare neurons with different baseline activities, they standardize the reading using the neuron's own spread on ordinary code (Set C). Crucially, this is done per "kind of position" (e.g., keyword vs. identifier at different points in a prompt) to account for positional bias.
- The Noise Unit: They draw the battery twice ("twins") to establish a noise floor, ensuring that any reported signal is significantly larger than what would happen by chance from a different set of random identifiers.
- The Licence: Before a profile is accepted, it must pass a "licence" of nine checks (see Table 2), including null-hypothesis tests and convergence checks.
- Readings Across Training: The process is repeated across hundreds of training checkpoints to track the "biography" of a concept.
Datasets
The method was realized for Python and tested on OpenCoder-1.5B.
- Reference Corpus: A public collection of 253,398 parsed Python files used to determine natural rates for identifiers, nesting, and companion constructs.
- Inventory: 188 Python constructs (grammar nodes and built-in names).
- Training History: The authors followed these constructs across 61 checkpoints, spanning two trillion training tokens.
Key Insights and Ideas
- Inventory vs. Grammar: One of the most striking findings is that when constructs share "carrier neurons," they cluster by inventory category (e.g., all built-in exception classes) rather than grammatical category (e.g., all "statements" or all "expressions"). The model groups things by what they are semantically, not just how they fit in a syntax tree.
- Probes Saturate, Profiles Don't: Linear probes (the standard tool for this) reach near-100% accuracy very early in training, making them "blind" to later refinements. Calibrated profiles, however, continue to shift and refine throughout the entire 2-trillion-token training run.
- Positional "Yardsticks": A neuron's behavior is highly dependent on its position. The authors found that using a single global mean/variance for standardization is often wrong; they instead use a 20-cell grid (token class $\times$ prompt length band) to provide a localized, accurate "yardstick."
Key Results
- Causal Validity: The authors performed ablation studies, zeroing out the "carrier neurons" identified by the profiles. This caused a significantly higher loss at the target construct's positions compared to zeroing out random sets of neurons, proving the identified neurons are functionally important.
- High Resolution: The measurement is sensitive enough to resolve changes between checkpoints just 1,000 training steps apart.
- Reliability: In the "licence" phase, the method showed high stability, with a median "twin agreement" (cosine similarity) of 0.986.
Limitations and Future Work
- Named Constructs Only: The primary limitation is that this method can only measure concepts you can already name and parse. It cannot "discover" new, hidden features like Sparse Autoencoders can.
- Distribution Dependence: Everything the profile says is relative to the "declared distribution." If the reference corpus doesn't accurately reflect the model's actual training data, the "ordinary" baseline might be skewed.
- Architecture Specificity: While the principles are general, the current implementation is tailored to transformer MLP blocks. Extending this to other architectures (like state-space models) would require redefining the "read site."
- Future Work: The authors suggest that a "battery that matches companions" and exploring the "natural arm" of re-weighting checks would be valuable next steps for the community.
Summary updated