Counterfactual Feature Steering for Adjacency-Preserving Identity Unlearning

Fnu Shivam, Layth Sliman, Ramana Reddy, Donald Adjeroh, Binod Bhattarai, Prashnna GyawaliarXiv 2026 · arXiv

TL;DR

Counterfactual Feature Steering (CFS) is a new inference-time method that surgically removes specific identities (like celebrities) from text-to-image models while protecting semantically related "adjacent" concepts. By using Sparse Autoencoders (SAEs) to perform bounded, per-token projections in latent space, CFS achieves a state-of-the-art balance between erasing target data and preserving the model's broader utility and related identities.

What Problem Does This Paper Address?

Text-to-image (T2I) models like Stable Diffusion are trained on massive, often uncurated web-scale datasets. This leads to a major privacy and legal headache: the models can memorize and reproduce the likenesses of real people without consent, potentially violating regulations like the EU's GDPR.

While "unlearning" methods exist to remove these concepts, they suffer from a "collateral damage" problem. If you try to unlearn a specific tech CEO, the model might accidentally suppress other CEOs or public figures who share similar visual features (suits, podiums, office settings). The authors call the ability to avoid this "adjacency preservation." Existing steering methods often apply a "one-size-fits-all" intervention strength that can cause the model's internal representations to collapse or fail to distinguish between a strong presence of a concept and a weak one.

Methods

The authors propose Counterfactual Feature Steering (CFS), which operates in the latent space of a Sparse Autoencoder (SAE) attached to the model's text encoder.

The process works in three main steps:

  1. Direction Identification: By contrasting the activations of "forget" prompts (those naming the target) and "retain" prompts (everything else), they identify a specific vector in the SAE's sparse feature space that represents the target identity.
  2. Alignment Measurement: At inference time, for every token in a prompt, CFS calculates an alignment score ($\alpha_t$). This measures how much that specific token "looks like" the target concept.
  3. Bounded Projection: Instead of simply subtracting a vector (which can "overshoot" and damage the model), CFS performs a projection. It moves the representation toward a "retain distribution" only as much as is needed to cancel out the target signal. This intervention is bounded—it never pushes the representation further than the "neutral" boundary, preventing the representation collapse seen in other methods.

Crucially, this is an inference-time hook. It doesn't require retraining the U-Net or storing massive weight-edited checkpoints; you just need the SAE and the pre-calculated target direction.

Datasets

The study utilizes several datasets to test different facets of unlearning:

  • Celeb-100: A corpus of 4,000 prompts covering 100 celebrities. The authors selected 8 diverse targets (e.g., Barack Obama, Elon Musk, Scarlett Johansson) for deep evaluation.
  • COCO-30K: 10,000 captions used to ensure that unlearning an identity doesn't degrade the model's general ability to generate high-quality images.
  • I2P (Inappropriate Image Prompts): Used to benchmark the method's ability to suppress nudity.
  • Unlearn Canvas: Used for artistic style removal (specifically targeting Van Gogh).
  • New Benchmark: The authors introduced the Adjacency-Weighted Unlearning Score (AWUS) and FaceAWUS (using ArcFace embeddings) to specifically quantify the tradeoff between target erasure and collateral damage to "neighboring" identities.

Key Insights and Ideas

  • The Power of Sparsity: The paper reinforces that SAEs are not just for "mechanistic interpretability" (understanding what a model does) but are highly effective tools for control. Operating in a sparse, monosemantic space allows for much more surgical edits than operating in dense latent spaces.
  • Boundedness Matters: Most steering methods are "unbounded"—if you turn the strength up too high, the image falls apart. CFS's projection-based approach is mathematically bounded, meaning it remains stable even at "full" intervention strength.
  • Token-Level Adaptivity: CFS is "input-adaptive." If a prompt only weakly mentions a concept, the intervention is light. If it's the central subject, the intervention is stronger. This prevents the "blunt instrument" effect of previous steering methods.

Key Results

  • Superior Tradeoff: CFS achieved the highest AWUS (0.296) and FaceAWUS (0.636) scores compared to baselines like ESD, UCE, and Concept Steerers. This means it erased the target more effectively while doing the least damage to peers (e.g., unlearning Obama while keeping Biden intact).
  • Stability: In "strength sweeps," CFS remained qualitatively and quantitatively stable at high intervention levels, whereas subtraction-based methods saw rapid degradation in image quality.
  • Generalization: While designed for identities, the same math worked for nudity suppression. On the I2P benchmark, CFS achieved a nudity detection rate of just 0.47%, significantly lower than the baseline SD 1.4 (17.8%) and other safety methods like SLD-MAX (1.74%).
  • Efficiency: CFS is incredibly fast. Because it only modifies the text encoder representation once per prompt, it adds virtually zero overhead to the generation process (9.50s per image vs. 9.52s for the unmodified model).

Limitations and Future Work

  • SAE Dependency: The method's success is entirely dependent on the quality of the underlying SAE. If the SAE fails to capture a specific feature or identity monosemantically, CFS cannot steer it effectively.
  • Not a "Hard" Security Fix: The authors note this is intended for model providers to control their hosted inference services. It is not a "tamper-resistant" mechanism; a user with full access to the weights and code could likely bypass the steering hook.
  • Benchmark Maturity: The authors highlight that "adjacency-preserving unlearning" is a new frontier, and more standardized, community-wide benchmarks are needed to evaluate these subtle tradeoffs properly.

Summary updated

Abstract

Identity unlearning in text-to-image diffusion models is increasingly important for privacy, safety, and ethical deployment, as it aims to remove the model's ability to generate specific individuals. In this work, we study identity unlearning with \emph{adjacency preservation}, where the goal is to remove a target individual's identity while preserving related non-target identities. We propose Counterfactual Feature Steering (CFS), a method requiring one-time SAE training that performs bounded inference-time concept editing in the latent space of a sparse autoencoder attached to the text encoder. CFS derives a target direction by contrasting forget and retain distributions, then applies a per-token projection proportional to each token's current alignment with this direction, yielding counterfactual representations that suppress the target while limiting collateral damage. To evaluate adjacency preservation, we introduce the Adjacency-Weighted Unlearning Score (AWUS), which measures the tradeoff between target erasure and collateral damage, and extend it to face-specific identity similarity using ArcFace. Across 4,000 prompts derived from Celeb-100, spanning 100 celebrity identities, CFS achieves the strongest evaluated tradeoff between identity erasure of target identities and preservation of adjacent and generic concepts, while remaining stable across intervention strengths where other steering-based methods degrade. Independent ArcFace evaluation and a 10,000-caption COCO study further examine identity and general preservation. Quantitative nudity-suppression benchmarks and qualitative artistic-style examples provide evidence of transfer beyond identities on SD~1.4.
BibTeX
@misc{shivam2026counterfactual,
  author = {FNU Shivam and Layth Sliman and Ramana Reddy and Donald Adjeroh and Binod Bhattarai and Prashnna Gyawali},
  title = {{Counterfactual Feature Steering for Adjacency-Preserving Identity Unlearning}},
  year = {2026},
  eprint = {2611.20637},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.20637},
}
Figure