Deep learning interpretability has traditionally focused on "input attribution"—identifying which pixels in an image or tokens in a sentence drive a model's prediction. However, understanding why a model arrives at a decision requires looking inside the "black box" to identify the specific internal units, or neurons, that process and store information. Gradient-based neuron attribution provides a mathematical framework to localize these internal mechanisms, moving from coarse layer-wise analysis to surgical, neuron-level insights.
The Intuition: Gradients as Sensitivity
The core idea behind neuron attribution is to treat internal activations as intermediate inputs. If we want to know how much a specific neuron in a hidden layer contributes to the final output, we can measure how sensitive the output is to changes in that neuron's activation.
In their work on person re-identification, Li et al. quantify this by calculating an "impact score"—the change in the loss function when a neuron is removed. Because exhaustively removing every neuron is computationally expensive, the authors approximate this impact using a first-order Taylor expansion. This approach uses the gradient of the loss with respect to the neuron's activation to estimate its importance. Similarly, Leng and Zhu et al. use Taylor expansions to identify "task-specific" and "language-aware" neurons, allowing for more efficient fine-tuning by updating only the most influential parameters.
Overcoming Saturation with Integrated Gradients
A simple gradient (the partial derivative of the output with respect to a neuron) can be misleading. In deep networks, neurons often "saturate"—once a feature is strongly detected, the gradient may drop to zero even though the neuron remains critical to the output.
To solve this, many researchers adapt Integrated Gradients (IG) for internal layers. Instead of looking at the gradient at a single point, IG calculates the integral of gradients along a path from a "neutral" baseline (like a zero activation) to the actual activation value. Dhamdhere et al. formalized this as conductance, a method that applies the chain rule to decompose input attributions into the flow of importance through specific hidden units.
This mechanism has proven essential for mapping knowledge in Large Language Models (LLMs). Ju et al. and Wang et al. use integrated gradients to locate "knowledge neurons" within feed-forward networks. By identifying these units, the authors can show how LLMs store factual associations and why they sometimes take "factual shortcuts" instead of performing multi-hop reasoning.
From Individual Neurons to Functional Circuits
While individual neurons can represent simple concepts, complex behaviors often emerge from groups of neurons working in sequence. Recent research has shifted toward identifying these "critical pathways."
- Joint Attribution: Wang et al. developed the Joint Attribution Score (JAS) to quantify the collective contribution of neurons across different layers in Vision Transformers. This reveals that models rely on sparse, semantically clustered paths rather than monolithic layers.
- Functional Interactions: Cao et al. introduced NeurFlow, which uses importance scores to isolate how a "target neuron" in one layer is influenced by a specific subset of neurons in the preceding layer, effectively mapping the hierarchical circuit of a concept.
- Relative Influence: Nam et al. argue that a neuron's importance is best understood relative to its neighbors. Their Relative Attributing Propagation (RAP) method filters out background noise by focusing on how much more a neuron contributes compared to the units surrounding it.
Surgical Interventions and Model Editing
The ultimate validation of neuron attribution is the ability to control a model by manipulating the identified units. If an attribution method correctly identifies the neurons responsible for a specific behavior, "muting" or "amplifying" those neurons should predictably change the model's output.
This has led to several "surgical" applications:
- Knowledge Editing: Liu et al. and Yang et al. use neuron-level attribution to update specific facts in LLMs. By targeting only the neurons responsible for a fact, they can edit the model's knowledge without causing the "catastrophic collapse" of its general reasoning abilities.
- Safety and Fairness: Zhao et al. identified that safety in LLMs is governed by less than 1% of neurons. By tuning these "safety neurons," they can reduce harmful outputs. Similarly, Zhai et al. use attribution to find and mute "biased" neurons that disproportionately rely on sensitive attributes like gender or race.
- Defense: In the realm of security, Fan et al. and Liang et al. use attribution to detect "backdoor" neurons—units that stay dormant during normal use but trigger malicious behavior when a specific pattern is present in the input.
By moving the focus from the input to the internal architecture, gradient-based neuron attribution provides a bridge between the mathematical weights of a network and the high-level concepts they represent.
Key Foundational Work
Learning Deep Feature Representations with Domain Guided Dropout for Person Re-identification (2016) — contributed 1 concept elements to this direction.
Visual Explanations for Convolutional Neural Networks via Input Resampling (2017) — contributed 1 concept elements to this direction.