For years, the standard way to understand a "black box" neural network was to look at its peaks. If you wanted to know what a specific neuron did, you would find the images or text snippets that made it fire most intensely. This "max-activating" heuristic led to the famous visualizations of "dog face" detectors or "sentiment" neurons.
However, a growing body of research suggests that these peaks are a misleading sliver of a model's actual logic. By focusing only on the highest activations, we ignore the vast majority of a neuron's causal impact, miss its "moonlighting" behaviors, and overlook the critical information hidden in negative signals.
The Two Percent Problem
The most striking critique of the max-activating approach is its lack of causal coverage. In Linear Explanations for Individual Neurons, researchers argue that the top fraction of activating inputs—the ones we usually use to label a neuron—often account for less than 2% of that neuron's total influence on the model's final output.
If we only explain the top 2%, we aren't explaining the neuron; we are explaining an outlier. This suggests that neurons are not simple "feature detectors" that are either "on" or "off." Instead, they act as high-dimensional filters where the mid-range and low-range activations carry the bulk of the computational load.
Neurons as Polysemantic Moonlighters
When we label a neuron based on its peak activation, we assume it represents a single, stable concept. But Laura Kopf and colleagues, through the PRISM framework, demonstrate that neurons are often "polysemantic"—they encode multiple, unrelated concepts simultaneously.
Crucially, these concepts may trigger the neuron at different intensities. A neuron might fire at 100% for "Golden Retrievers" but at 40% for "furry textures" and 10% for "yellow objects." As Biagio La Rosa shows in Towards a fuller understanding of neurons with Clustered Compositional Explanations, neurons often "moonlight," changing their functional role entirely depending on their activation level. Suppressing these non-peak signals can alter model predictions in over 10% of cases, proving they aren't just "noise."
The Importance of Negative Space
Most interpretability methods, inherited from the era of ReLUs (which zero out negative values), ignore negative pre-activations. However, modern models use smooth functions like GELU or SiLU, where negative values are mathematically possible and computationally relevant.
Linghao Kong and Angelina Ning discovered that these "negative" regions are actually the primary substrate for processing syntax. In Negative Pre-activations Differentiate Syntax, they show that perturbing just 1% of these negative signals can destroy a model's grammatical ability while leaving its reasoning intact. Similarly, Nicholas Bai argues in the Describe-and-Dissect (DnD) framework that ignoring inhibitory (negative) signals provides an incomplete profile of a neuron's role.
Moving Toward Holistic Explanations
To fix these blind spots, researchers are moving toward more expressive, automated explanation methods:
- Sparse Autoencoders (SAEs): Instead of looking at raw neurons, researchers like Charles O'Neill use SAEs to disentangle messy activations into "features." However, even SAEs can be biased by top-activating examples. George Ma suggests we must actively "falsify" these explanations by checking if they hold up at lower activation thresholds.
- Concept Arithmetic: Ruth Fong and Andrea Vedaldi show in Net2Vec that concepts are distributed across many filters. We can't understand "grass" by looking at one filter; we have to look at how multiple filters combine.
- LLM-Based Summarization: Frameworks like SASC (Chandan Singh et al.) and the work by Gonçalo Paulo use Large Language Models to generate natural language descriptions that cover a wider range of a feature's behavior, moving beyond simple labels to complex, nuanced explanations.
The shift in the field is clear: to truly "open the black box," we must stop looking only at the brightest lights and start examining the shadows and mid-tones of neural activity. Only then can we build a robust, causal map of how these models actually think.