Calibrated concept profiles: making the reading of a model's neurons a measurement

Piotr WilamarXiv 2026 · arXiv

TL;DR

The paper introduces calibrated concept profiles, a framework that transforms direct neuron reading from a subjective inspection into a rigorous, unit-based measurement. By using controlled stimulus distributions and standardized units, it allows researchers to track how specific concepts (like Python for loops) emerge and evolve across a model's entire training history without needing to train secondary models like sparse autoencoders.

What Problem Does This Paper Address?

In the quest to understand the "black box" of Large Language Models (LLMs), the field of interpretability has largely moved away from looking at individual neurons. Direct reading was plagued by "polysemanticity" (neurons having multiple meanings), a lack of standardized units, and results that didn't transfer between studies. Consequently, researchers turned to "second models"—like linear probes or sparse autoencoders (SAEs)—to interpret the first.

However, this move comes at a cost: you lose the ability to name the model's native units directly and perform simple causal tests (like ablating a specific neuron). This paper argues that direct neuron reading isn't fundamentally flawed; it just lacks the rigor of a formal measurement. The authors aim to bring that rigor back by defining clear populations of stimuli, standardized units, and validation "licences."

Methods

The authors propose a six-step process to create a calibrated concept profile for any given construct:

  1. Prompt Generator: Instead of hand-writing a few examples, they generate a "battery" of prompts based on a declared distribution measured from a real-world reference corpus. They create three sets: Set A (the target concept in its role), Set B (the same token but in a control role, like inside a comment), and Set C (baseline ordinary code).
  2. Raw Readings: They calculate the difference in mean activation between Set A and Set B. This isolates the neuron's response to the role of the concept, effectively subtracting the "noise" of the token itself.
  3. The Profile (Standardization): To compare neurons with different baseline activities, they standardize the reading using the neuron's own spread on ordinary code (Set C). Crucially, this is done per "kind of position" (e.g., keyword vs. identifier at different points in a prompt) to account for positional bias.
  4. The Noise Unit: They draw the battery twice ("twins") to establish a noise floor, ensuring that any reported signal is significantly larger than what would happen by chance from a different set of random identifiers.
  5. The Licence: Before a profile is accepted, it must pass a "licence" of nine checks (see Table 2), including null-hypothesis tests and convergence checks.
  6. Readings Across Training: The process is repeated across hundreds of training checkpoints to track the "biography" of a concept.
Figure 1: The calibrated concept profile in six steps (§2.1–2.6), for one construct, at each checkpoint.

Datasets

The method was realized for Python and tested on OpenCoder-1.5B.

  • Reference Corpus: A public collection of 253,398 parsed Python files used to determine natural rates for identifiers, nesting, and companion constructs.
  • Inventory: 188 Python constructs (grammar nodes and built-in names).
  • Training History: The authors followed these constructs across 61 checkpoints, spanning two trillion training tokens.

Key Insights and Ideas

  • Inventory vs. Grammar: One of the most striking findings is that when constructs share "carrier neurons," they cluster by inventory category (e.g., all built-in exception classes) rather than grammatical category (e.g., all "statements" or all "expressions"). The model groups things by what they are semantically, not just how they fit in a syntax tree.
  • Probes Saturate, Profiles Don't: Linear probes (the standard tool for this) reach near-100% accuracy very early in training, making them "blind" to later refinements. Calibrated profiles, however, continue to shift and refine throughout the entire 2-trillion-token training run.
  • Positional "Yardsticks": A neuron's behavior is highly dependent on its position. The authors found that using a single global mean/variance for standardization is often wrong; they instead use a 20-cell grid (token class $\times$ prompt length band) to provide a localized, accurate "yardstick."

Key Results

  • Causal Validity: The authors performed ablation studies, zeroing out the "carrier neurons" identified by the profiles. This caused a significantly higher loss at the target construct's positions compared to zeroing out random sets of neurons, proving the identified neurons are functionally important.
  • High Resolution: The measurement is sensitive enough to resolve changes between checkpoints just 1,000 training steps apart.
  • Reliability: In the "licence" phase, the method showed high stability, with a median "twin agreement" (cosine similarity) of 0.986.
Table 4: The licence on OpenCoder-1.5B, final checkpoint.

Limitations and Future Work

  • Named Constructs Only: The primary limitation is that this method can only measure concepts you can already name and parse. It cannot "discover" new, hidden features like Sparse Autoencoders can.
  • Distribution Dependence: Everything the profile says is relative to the "declared distribution." If the reference corpus doesn't accurately reflect the model's actual training data, the "ordinary" baseline might be skewed.
  • Architecture Specificity: While the principles are general, the current implementation is tailored to transformer MLP blocks. Extending this to other architectures (like state-space models) would require redefining the "read site."
  • Future Work: The authors suggest that a "battery that matches companions" and exploring the "natural arm" of re-weighting checks would be valuable next steps for the community.

Summary updated

Abstract

Interpretability has largely moved from reading a model's neurons directly to reading a second model trained on the first: a probe, a sparse autoencoder, a crosscoder. The move answered real faults -- mixed meanings, thresholds without a unit, readings that did not transfer from one study to the next -- but gave up what a direct reading alone provides: nothing trained in between, the neurons themselves named, and a reading that can be repeated and compared. Here the direct reading is made a measurement. A neuron's activation varies with the prompt, so its reading is a distribution over a population of stimuli. We declare that population: every factor around a target construct is drawn at its rate in a reference corpus and logged. From the declared distribution we generate a battery of prompts, and draw it twice. From the battery we compute the calibrated concept profile, one number per neuron, stated in units of the neuron's own spread on ordinary input; the two draws give every number a noise unit. Before the profile is read, a fixed set of checks licenses the battery on the model, and the smallest change the profile can resolve across training checkpoints is measured. Realised for Python on OpenCoder-1.5B and followed across checkpoints spanning two trillion training tokens, the profile gives what the alternatives do not report: a unit on every number, a chance level for every comparison, and which neurons carry a construct at every checkpoint. The battery, the code and the data behind every result are released.
BibTeX
@misc{wilam2026calibrated,
  author = {Piotr Wilam},
  title = {{Calibrated concept profiles: making the reading of a model's neurons a measurement}},
  year = {2026},
  eprint = {2611.09196},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.09196},
}
Figure