When a language model makes a prediction—whether it is classifying the sentiment of a financial report, generating the next word in a medical summary, or retrieving a document—it operates over thousands or millions of parameters. To humans, the model is a black box. We want to know: Which specific words or tokens caused this output?
While it is tempting to look at internal mechanisms like attention weights to answer this question, attention can be a misleading indicator of true feature importance. As Sinan Fan demonstrates in the context of soft prompt optimization, attention weights alone often fail to diagnose how specific tokens distract a model and degrade its reasoning performance (Improving Complex Reasoning with Dynamic Prompt Corruption: A Soft Prompt Optimization Approach).
Instead, a more mathematically rigorous approach relies on calculus. Gradient-based feature attribution measures the sensitivity of a model's output to its inputs. By calculating the derivative of a target output with respect to continuous token embeddings, we can map abstract internal activations back to discrete, human-readable words.
The Core Intuition: Gradients as Sensitivity
Consider a neural network that takes a sequence of tokens as input, maps them to continuous embedding vectors $x_1, x_2, \dots, x_n$, and outputs a scalar prediction score $y$ (such as the logit for a specific class).
The partial derivative $\frac{\partial y}{\partial x_i}$ tells us how much the output $y$ would change if we made an infinitesimal change to the embedding of token $i$. If the gradient is large, the model's output is highly sensitive to that token; if it is zero, changing the token has no immediate effect on the prediction.
To turn the vector-valued gradient $\frac{\partial y}{\partial x_i}$ into a single scalar importance score for a token, researchers often compute the dot product between the token's embedding and its gradient ($x_i \cdot \nabla_{x_i} y$) or take its L2 norm. This basic technique, known as saliency mapping, was used by Sweta Karlekar et al. to interpret neural models trained on speech transcripts (Detecting Linguistic Characteristics of Alzheimer's Dementia by Interpreting Neural Models). By computing the derivative of the model's output with respect to input embeddings, the authors were able to visualize the internal decision-making process and isolate high-impact linguistic markers.
Overcoming Saturation with Path-Based Attribution
While vanilla gradients offer a computationally efficient snapshot of sensitivity, they suffer from a fundamental flaw known as saturation.
Imagine a model that triggers an alarm if a text contains the word "danger." Once the embedding for "danger" is prominent enough that the model is 99.99% confident, the output logit flattens out. At this point, increasing the intensity of the feature further barely changes the output, meaning the local gradient drops to zero. A simple saliency map would mistakenly conclude that the word "danger" has no influence on the prediction.
To resolve this, path-based attribution methods like Integrated Gradients (IG) are used. Instead of evaluating the gradient only at the final input, IG defines a path from a baseline "null" input (such as an all-zero embedding sequence) to the actual input. It samples the gradients at intervals along this linear path and integrates them.
Baseline Input (e.g., [PAD]) ---------> Intermediate States ---------> Actual Input Text
Gradient = High Gradient = Saturating Gradient = Zero
|---------------------------------- Integrated Gradients ---------------------------------|
By accumulating gradients along the entire trajectory, IG captures the feature's impact before the model saturates. For example, Pramod Kaushik Mudrakarta et al. utilized Integrated Gradients to reveal that question-answering models frequently ignore critical question tokens, allowing attackers to plummet model accuracy by adding content-free prefixes (Did the Model Understand the Question?). Similarly, Kwabena Amponsah-Kaakyire et al. used IG to audit translation-detection models, proving whether a classifier relies on genuine syntactic patterns or merely exploits spurious correlations like topic-specific keywords and punctuation (Explaining Translationese: why are Neural Classifiers Better and what do they Learn?).
Choosing the Target Layer: Input vs. Hidden States
Gradients can be calculated with respect to any continuous representation inside a transformer network. The choice of layer determines what kind of intuition we gain about the model's internal logic.
1. Input Embeddings
Computing gradients at the initial embedding layer ties the attribution directly to the input tokens. Chih-Kuan Yeh et al. demonstrate that when tracking data influence (how much a training example affects a test prediction), computing gradients at the first layer avoids a "cancellation effect" that plagues deeper layers (First is Better Than Last for Language Data Influence). This word-level decomposition allows developers to pinpoint exactly which training words drive a specific test-time response.
2. Intermediate Hidden Layers
Alternatively, calculating gradients with respect to intermediate hidden states reveals how abstract concepts are processed across layers. Hosein Mohebbi et al. applied gradient-based token attribution to diagnostic classifiers attached to BERT's hidden layers (Exploring the Role of BERT Token Representations to Explain Sentence Probing Results). This approach allowed them to move beyond layer-level performance metrics and discover that BERT anchors specific linguistic features (like grammar or sentence length) in distinct "playmaker" tokens or punctuation marks.
Auditing Vulnerabilities, Hallucinations, and Safety
Because gradient attribution exposes the exact features driving a model's behavior, it serves as a primary tool for auditing model safety, identifying biases, and defending against adversarial manipulation.
Adversarial Vulnerabilities and Shortcuts
Attackers can use gradients to find a model's weak points. Jinfeng Li et al. developed TextBugger, a framework that computes the Jacobian matrix of a model's output to isolate tokens with the highest gradient influence (TextBugger: Generating Adversarial Text Against Real-world Applications). By targeting these high-gradient words for character- or word-level swaps, they can efficiently break commercial text classifiers.
On the auditing side, Somesh Singh used Integrated Gradients to show that automated essay scoring systems often operate like simple bag-of-words models, drastically altering scores based on a few high-gradient trigger words rather than evaluating actual contextual coherence (Automatic Essay Scoring Systems Are Both Overstable And Oversensitive: Explaining Why And Proposing Defenses). Similarly, Nicola Fanton et al. used IG to track how language models rely on demographic stereotypes over time when predicting the target audience of online documents (A Diachronic Analysis of Human and Model Predictions on Audience Gender in How-to Guides).
Diagnosing Generative Models
In generative tasks, gradients help trace where information originates. For Retrieval-Augmented Generation (RAG), Jessica Hoffmann showed that calculating the gradient of a generated token's probability with respect to source text embeddings provides a white-box mechanism to verify if a generated claim is factually grounded in the context (Detecting Hallucination and Coverage Errors in Retrieval Augmented Generation for Controversial Topics).
Gradients are equally effective for diagnosing data leakage in text-to-image generation. Yuxin Wen et al. tracked memorization in diffusion models by taking the gradient of the text-conditional noise prediction magnitude with respect to prompt token embeddings (Detecting, Explaining, and Mitigating Memorization in Diffusion Models). This significance scoring automatically isolates the exact linguistic triggers that force a model to reproduce verbatim training samples.
Active Security Defenses
Instead of just explaining failures, gradient attribution can actively mitigate them. Jiazhao Li et al. designed AttDef, a pipeline that calculates token-level attribution scores to identify and mask malicious backdoor triggers before they can influence a model's prediction (Defending against Insertion-based Textual Backdoor Attacks via Attribution).
In a similar vein, San Kim developed GMTP to protect RAG pipelines from corpus poisoning (Safeguarding RAG Pipelines with GMTP: A Gradient-based Masked Token Probability Method for Poisoned Document Detection). By computing the gradient of a retriever's similarity function with respect to individual token embeddings, GMTP isolates the specific strings responsible for artificially inflating a document's rank, allowing the system to filter out adversarial text with minimal latency.
From Passive Interpretation to Active Regularization
The ultimate utility of gradient attribution extends beyond post-hoc analysis; gradient signals can be integrated directly into training objectives to alter how a model learns.
For data augmentation, Soyoung Yoon introduced SSMix, a text classification strategy that uses gradient-based saliency mapping to identify the most and least informative spans in a pair of sentences (SSMix: Saliency-Based Span Mixup for Text Classification). By replacing the non-salient tokens of a target sentence with the highly salient tokens of a source sentence, the model creates synthetic training examples that force it to focus on robust semantic features.
Sentence A (Target): "The movie was [not bad], but the seating was cramped."
| |
Low Saliency High Saliency
| |
Sentence B (Source): "An [excellent] cinematic masterpiece."
|
High Saliency
v
Synthetic Mixup: "The movie was [excellent], but the seating was cramped."
To defend against adversarial word-swapping, Pengwei Zhan et al. developed SIWCon (Similarizing the Influence of Words with Contrastive Learning to Defend Word-level Adversarial Text Attack). SIWCon uses gradient-based attribution to rank token importance and applies a contrastive loss penalty during training. This regularizes the network, forcing it to distribute its attention and influence more evenly across the entire sentence rather than over-relying on a few fragile, high-gradient keywords.
By shifting from passive interpretation to active regularization, gradient-based feature attribution transforms from a simple diagnostic lens into a foundational mechanism for building robust, secure, and trustworthy language models.