Beyond Input Activations: Identifying Influential Latents by Gradient Sparse Autoencoders

Dong Shu, Xuansheng Wu, Haiyan Zhao, Mengnan Du, Ninghao Liu

Abstract

Sparse Autoencoders (SAEs) have recently emerged as powerful tools for interpreting and steering the internal representations of large language models (LLMs). However, conventional approaches to analyzing SAEs typically rely solely on input-side activations, without considering the causal influence between each latent feature and the model's output. This work is built on two key hypotheses: (1) activated latents do not contribute equally to the construction of the model's output, and (2) only latents with high causal influence are effective for model steering. To validate these hypotheses, we propose Gradient Sparse Autoencoder (GradSAE), a simple yet effective method that identifies the most influential latents by incorporating output-side gradient information.

Introduction

Large Language Models (LLMs) are often viewed as "black boxes." Sparse Autoencoders (SAEs) attempt to open this box by decomposing complex neural activity into interpretable "latents"—features like "legal jargon" or "geographic references." Traditionally, researchers have assumed that if a latent is active, it is important for the model's current output.

However, this assumption is challenged by the phenomenon of superposition, where models pack numerous features into a limited number of neurons. Consequently, many latents may activate for a given input, but only a small subset actually drives the final response. Relying solely on input activations leads to imprecise model steering and "output collapse" when irrelevant but active latents are modified.

Methodology: The GradSAE Framework

To address the gap between activation and causal influence, we introduce GradSAE (Gradient Sparse Autoencoder). GradSAE identifies influential latents by incorporating the gradient of the output probability with respect to each latent.

The GradSAE framework calculates influence scores by multiplying latent activations by output-side gradients.

Causal Influence via Gradient Approximation

Ideally, identifying an influential latent requires ablation—setting a latent to zero and measuring the change in output. Given the computational cost of brute-force ablation across thousands of latents, GradSAE uses a first-order Taylor expansion to approximate this effect. The influence score ($g$) is calculated as the element-wise product of the gradient and the activation: $$g \approx \text{Gradient} \times \text{Activation}$$

This approach is training-free and "plug-and-play," requiring only a single backward pass to calculate gradients without retraining the LLM or the SAE.

Experimental Setup

We evaluated GradSAE using the SQuAD (Stanford Question Answering Dataset) on Gemma 2 9B and LLaMA 3 8B models. SQuAD is particularly useful because it provides multiple questions for the same context, enabling "local steering" experiments where we attempt to force the model to answer a target question while processing a different one.

Key Results and Insights

Our analysis reveals that Activation ≠ Importance. Only approximately 18% of the most active latents are also among the most influential. Gradients serve as a "relevance filter," isolating the features that subsequent layers actually utilize to generate a response.

Perturbation and Steering Performance

GradSAE significantly outperforms activation-based baselines in both identifying critical features and steering model behavior.

Comparative performance of GradSAE vs. baseline across perturbation and local steering tasks.
  • Precision in Perturbation: Masking the single most influential latent identified by GradSAE dropped the model's Exact Match (EM) score from 100% to 80.45%, whereas masking the most active latent only reduced it to 94.79%.
  • Superior Steering: In steering tasks, GradSAE-selected latents were more effective. At $K=10$, GradSAE achieved a steering F1 score of 10.21%, compared to 6.69% for the baseline.
  • Stability: Zeroing out the "BottomK" (least influential) latents had negligible impact on performance, even if those latents had high activation values, confirming that much latent activity is task-irrelevant noise.

Conclusion and Future Work

GradSAE demonstrates that causal influence, rather than mere activation magnitude, is the key to precise LLM interpretation and steering. While the backward pass introduces some computational overhead, the gains in precision are substantial. Future research will explore the application of GradSAE to pre-trained models and more complex, creative generation tasks beyond factual question-answering.

Summary updated

Abstract

Sparse Autoencoders (SAEs) have recently emerged as powerful tools for interpreting and steering the internal representations of large language models (LLMs). However, conventional approaches to analyzing SAEs typically rely solely on input-side activations, without considering the causal influence between each latent feature and the model's output. This work is built on two key hypotheses: (1) activated latents do not contribute equally to the construction of the model's output, and (2) only latents with high causal influence are effective for model steering. To validate these hypotheses, we propose Gradient Sparse Autoencoder (GradSAE), a simple yet effective method that identifies the most influential latents by incorporating output-side gradient information.
BibTeX
@misc{shu2025beyond,
  author = {Dong Shu and Xuansheng Wu and Haiyan Zhao and Mengnan Du and Ninghao Liu},
  title = {{Beyond Input Activations: Identifying Influential Latents by Gradient Sparse Autoencoders}},
  year = {2025},
  eprint = {2505.08080},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2505.08080},
}
Figure