A Single Neuron Works: Precise Concept Erasure in Text-to-Image Diffusion Models

Qinqin He, Jiaqi Weng, Jialing Tao, Hui XuearXiv 2025 · arXiv · OpenReview

TL;DR

This paper introduces SNCE, a novel method for precisely removing harmful concepts from text-to-image models by targeting individual neurons. By leveraging Sparse Autoencoders (SAEs) to disentangle concepts into atomic semantic units, SNCE achieves state-of-the-art concept erasure with minimal degradation to image quality and strong robustness against adversarial attacks.

What Problem Does This Paper Address?

Text-to-image diffusion models, while incredibly powerful for generating realistic visuals, also present significant safety concerns. They can inadvertently produce harmful content, such as explicit material, violence, or dangerous objects. Existing methods to mitigate this, like content filtering or coarse-grained model modifications, often struggle with precision. They can either be ineffective, require extensive retraining, or, crucially, degrade the quality of generated images by impacting unrelated concepts. The challenge lies in surgically removing specific unwanted concepts without compromising the model's overall generative capabilities and image fidelity.

Methods

The core of SNCE lies in its ability to pinpoint and manipulate individual neurons responsible for specific concepts. The approach involves three main steps:

  1. Sparse Autoencoder (SAE) Training: An SAE is trained on text encoder activations. SAEs are known for their ability to decompose dense representations into sparse, interpretable latent spaces where individual neurons often correspond to atomic semantic concepts. The authors specifically use a "TopK SAE" variant, which ensures a fixed number of active neurons, leading to more predictable sparsity.
  2. Concept-Specific Neuron Identification: This is a novel contribution. The authors devise a "modulated frequency scoring" method. They construct "concept-pairs" (e.g., "a photo of a naked woman" vs. "a photo of a woman") and analyze the SAE activations. Neurons that are highly activated by the "concept" prompt but not by the "deconcept" prompt are identified as concept-specific. These neurons are then ranked by their weighted frequency scores.
  3. Targeted Concept Manipulation: Once the top-k concept-specific neurons are identified, their activations are suppressed during the text-to-image generation process. This is done by applying a manipulation mask and a coefficient ($\lambda$) that controls the intensity of suppression. This fine-grained intervention allows for precise removal of the target concept.

Datasets

  • Training: The authors use a combination of the DiffusionDB dataset (prompts with NSFW scores > 0.5) and the Inappropriate Image Prompts (I2P) dataset, totaling 30,000 text samples.
  • Evaluation:
    • Nudity Concept Erasure: Evaluated on the I2P benchmark and MS COCO-30K.
  • Violence Concept Erasure: Evaluated on the I2P dataset.
  • Adversarial Robustness: Evaluated on the P4D (nudity subset) and Ring-A-Bell (nudity subset) datasets.
  • Benign Content Preservation: Evaluated on the MS COCO-30K dataset.

Key Insights and Ideas

  • Single Neuron Power: The central insight is that a single neuron can be highly responsible for generating a specific, potentially harmful, concept. This allows for extremely precise interventions.
  • SAEs for Disentanglement: Sparse Autoencoders are crucial for breaking down complex text embeddings into atomic, interpretable semantic units, making concept localization feasible.
  • Concept-Pairing for Specificity: The clever use of "concept-pairs" is key to filtering out general activations and isolating neurons that are truly specific to the target concept.
  • Fine-grained Control: SNCE offers a scalable way to control the intensity of concept erasure through the manipulation coefficient ($\lambda$) and the selection of top-k neurons, allowing for a trade-off between erasure strength and image quality.

Key Results

  • State-of-the-Art Erasure: SNCE achieves the lowest total nudity detection counts across various methods, showing over 50% improvement compared to some baselines.
  • Excellent Quality Preservation: The method maintains high FID scores (e.g., 15.85 for top-1 neuron manipulation) and competitive CLIP scores (e.g., 30.97), indicating minimal degradation of image quality and semantic alignment.
  • Robustness: SNCE demonstrates strong robustness against adversarial prompts, achieving significantly lower Attack Success Rates (ASR) compared to existing methods on benchmarks like P4D and Ring-A-Bell. For example, it achieved an ASR of 42.6% on P4D, outperforming UCE (80.2%) and SLD (77.5%).
  • Scalability: The method is shown to be applicable to various concepts (nudity, violence, guns, knives) and can be tuned via the manipulation coefficient $\lambda$ and the number of selected neurons.

Limitations and Future Work

  • Computational Cost: While SNCE is efficient, training SAEs and identifying neurons still requires computational resources.
  • Generalizability to All Concepts: While tested on several safety-related concepts, its performance on a broader range of concepts might require further investigation.
  • Future Work: The authors suggest exploring the application of SNCE to other generative models and investigating more complex concept interactions. The scalable top-k selection mechanism also offers avenues for further optimization and fine-tuning.

Summary updated

Abstract

Text-to-image models exhibit remarkable capabilities in image generation. However, they also pose safety risks of generating harmful content. A key challenge of existing concept erasure methods is the precise removal of target concepts while minimizing degradation of image quality. In this paper, we propose Single Neuron-based Concept Erasure (SNCE), a novel approach that can precisely prevent harmful content generation by manipulating only a single neuron. Specifically, we train a Sparse Autoencoder (SAE) to map text embeddings into a sparse, disentangled latent space, where individual neurons align tightly with atomic semantic concepts. To accurately locate neurons responsible for harmful concepts, we design a novel neuron identification method based on the modulated frequency scoring of activation patterns. By suppressing activations of the harmful concept-specific neuron, SNCE achieves surgical precision in concept erasure with minimal disruption to image quality. Experiments on various benchmarks demonstrate that SNCE achieves state-of-the-art results in target concept erasure, while preserving the model's generation capabilities for non-target concepts. Additionally, our method exhibits strong robustness against adversarial attacks, significantly outperforming existing methods.
BibTeX
@misc{he2025single,
  author = {Qinqin He and Jiaqi Weng and Jialing Tao and Hui Xue},
  title = {{A Single Neuron Works: Precise Concept Erasure in Text-to-Image Diffusion Models}},
  year = {2025},
  eprint = {2509.21008},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2509.21008},
}
Figure