Beyond Fixed Anchors: Precisely Erasing Concepts with Sibling Exclusive Counterparts

Tong Zhang, Ru Zhang, Jianyi Liu, Zhen Yang, Gongshen LiuarXiv 2025 · DOI · arXiv

Concept erasure is the "surgical" removal of specific ideas—such as nudity, copyrighted characters, or specific artistic styles—from text-to-image diffusion models. While essential for safety and ethics, current methods are often either "leaky" (the concept re-emerges under certain prompts) or "destructive" (they accidentally degrade related, safe concepts).

A new framework called SELECT (Sibling-Exclusive Evaluation for Contextual Targeting) addresses these flaws by moving away from rigid, one-size-fits-all replacement targets in favor of dynamic, semantically-aware anchors.

The Problem with Fixed Anchors

Traditional erasure methods typically map a target concept (e.g., "nudity") to a fixed anchor (e.g., "clothed person"). This approach suffers from two major issues:

  1. Concept Re-emergence: The target concept often leaks back into generated images, especially when triggered by complex or adversarial prompts.
  2. Concept Erosion: The model becomes "confused," leading to the degradation of neighboring concepts (e.g., erasing "Corgi" might inadvertently ruin the model's ability to generate "Beagles").

Methodology: Understanding the Model's Latent Space

The researchers first used causal tracing to visualize where concepts reside within the neural network. They found that abstract concepts (like art styles) are dispersed across many layers, while specific instances (like celebrities) are concentrated in just a few. This distribution proves that a single fixed anchor cannot effectively handle the diversity of concepts stored in the model.

Figure 2: Causal intervention analysis for a "Corgi" instance, showing how restoring specific layers re-introduces the erased concept.

The SELECT Framework

SELECT introduces a three-step process to find the optimal "Sibling Exclusive Concept" (SEC) to act as an anchor for erasure:

  1. Candidate Generation: An LLM generates a list of SECs—concepts that share a "parent" category (e.g., "cat" and "dog" are both "pets") but have mutually exclusive features.
  2. Two-Stage Evaluation:
  3. Contextual Activation: The framework measures how strongly a context triggers the target concept and selects anchors that minimize this residual activation.
  4. Semantic Coherence: It ensures the replacement anchor fits naturally into the prompt (e.g., replacing a "cat" with a "raccoon" in a forest scene is more coherent than replacing it with a "car").
  5. Anchor-Guided Retain: To prevent erosion, the framework identifies "boundary concepts" (other siblings) and explicitly protects them during the erasure process.
Figure 5: Evaluation of various candidate anchors for erasing "cat" and "Pikachu." Green checks indicate successful sibling exclusive concepts that prevent leakage while maintaining coherence.

Key Insights and Results

  • The "Smooth Path" Principle: Effective erasure requires a "smooth" semantic transition. Mapping a "cat" to a "raccoon" provides a more natural transition for the model's weights than mapping it to an unrelated object like a "car," which often causes visual artifacts.
  • Superior Performance: SELECT consistently outperformed baselines (like MACE and SPEED) across object, celebrity, artistic style, and NSFW erasure tasks. It achieved the lowest erasure error and the highest content retention.
  • Efficiency and Compatibility: The anchor mining process is remarkably fast, averaging only 4 seconds per concept. Furthermore, SELECT is a universal solution that can be integrated into existing fine-tuning or closed-form erasure algorithms.

Conclusion

By shifting from static anchors to dynamic, context-aware siblings, SELECT provides a more precise and less destructive way to manage the content of diffusion models. Future work will explore how these dynamic anchors scale to thousands of simultaneous concepts and how they behave in increasingly complex multi-concept prompts.

Summary updated

BibTeX
@misc{zhang2025beyond,
  author = {Tong Zhang and Ru Zhang and Jianyi Liu and Zhen Yang and Gongshen Liu},
  title = {{Beyond Fixed Anchors: Precisely Erasing Concepts with Sibling Exclusive Counterparts}},
  year = {2025},
  eprint = {2510.16342},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2510.16342},
}
Figure