Causal Feature Steering with Sparse Autoencoders

This direction is merged into another direction: sparse feature decomposition. Its last version stays readable and citable here.

Version 1 of 162 papers69 conceptsfrozen 2026-10-04

Neural networks are notoriously difficult to interpret because their internal representations are "polysemantic." A single neuron in a Large Language Model (LLM) or a Vision Transformer might activate for a wide variety of unrelated concepts—a phenomenon often attributed to the model packing more features into its hidden layers than it has dimensions, a state known as superposition.

To solve this, researchers have converged on a technique called Causal Feature Steering. By using Sparse Autoencoders (SAEs) to "refactor" these messy internal states into a library of discrete, human-understandable features, we can move beyond merely observing what a model does to actively controlling how it thinks.

Decomposing the Black Box

The core mechanism relies on training a secondary model—the Sparse Autoencoder—to reconstruct the activations of a frozen base model. As Alex Tamkin, Noah Goodman, and their colleagues demonstrate in Codebook Features, this process forces the model’s continuous internal states into a sparse set of "codes."

Instead of a dense vector where every number is slightly active, an SAE identifies a basis where only a few "features" are active at any given time. These features often correspond to specific semantic concepts, such as "legal jargon," "French grammar," or "circular objects." In InterPLM, Elana P. Simon applied this to protein language models, uncovering hidden biological features that dictate protein structure and function.

From Correlation to Causality

A major pitfall in interpretability is mistaking correlation for causation. A feature might activate whenever a model sees a "garden path" sentence, but that doesn't mean the feature is driving the model's interpretation. Abhinav Menon and David Krueger highlight this in Analyzing (In)Abilities of SAEs via Formal Languages, noting that many "interpretable" features have zero impact on the model's actual computation.

To verify a feature's functional role, researchers use causal interventions. This involves three primary techniques:

  1. Ablation: Zeroing out a feature to see if the model loses a specific ability (e.g., the ability to conjugate verbs).
  2. Clamping/Scaling: Manually setting a feature to a high value to force a behavior (e.g., forcing a model to adopt a specific persona).
  3. Patching: Replacing the activations of one prompt with those of another to see if the model's logic follows the "patched" feature.

Fazl Barez and Neel Nanda utilized these interventions in Scaling sparse feature circuit finding to map the "circuits" responsible for in-context learning, proving that specific sparse features are the primary drivers of the model's ability to learn from examples in a prompt.

Steering Across Modalities

The power of causal steering is not limited to text. In the realm of computer vision, Samuel Stevens and Yu Su showed in Interpretable and Testable Vision Features via Sparse Autoencoders that scaling sparse activations can predictably alter image classification and segmentation.

Similarly, Viacheslav Surkov and Robert West demonstrated in One-Step is Enough that SAEs can decompose the U-net representations of diffusion models. By manually adjusting these "conceptual knobs," they could steer image attributes like style, composition, and texture without retraining the underlying model.

Safety and Alignment Applications

Perhaps the most critical application of this research is in AI safety. Jeffrey George Wang explored how "persona features" control emergent misalignment in Persona Features Control Emergent Misalignment. By identifying the internal features that represent a "toxic persona," researchers can surgically suppress these activations to prevent harmful outputs.

This approach also extends to reward models. In SAFER, Xiang Wang demonstrated that intervening on sparse features allows for precise modulation of a model's safety scores, enabling researchers to "denoise" safety data or detect when a model is being "sycophantic" to an evaluator, as discussed by David D. Baek and Max Tegmark in Sycophancy Towards Researchers Drives Performative Misalignment.

Why It Works: The Sparse Bottleneck

Causal steering works because it exploits the model's own internal logic. By training an SAE with a sparsity constraint (often an L1 penalty), we force the autoencoder to find the most efficient "dictionary" of concepts that the base model is using.

When we intervene on these features, we aren't just adding noise; we are communicating with the model in its own internal language. As Sophie Hao and Tal Linzen found in their study of Verb Conjugation in Transformers, even complex linguistic rules are often encoded in simple linear subspaces. Flipping a vector across these subspaces allows for near-total control over the model's categorical outputs, turning the "black box" into a transparent, steerable machine.

Papers in this version

Showing 60 of 62 papers.

History & citationVersion 1 · 2026-10-04

Merged into another direction · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_causal_feature_steering_with_sparse_auto_263cc979,
  title = {Causal Feature Steering with Sparse Autoencoders},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/causal-feature-steering-with-sparse-autoencoders/rdr_263cc97916554d4e80073a6bfd224e2f},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-21},
}

This exact version (version 1, frozen text):

@misc{lacuna_causal_feature_steering_with_sparse_auto_263cc979_v1,
  title = {Causal Feature Steering with Sparse Autoencoders},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/causal-feature-steering-with-sparse-autoencoders/rdr_263cc97916554d4e80073a6bfd224e2f/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-21},
}