Abstract
As text-to-image diffusion models (DMs) proliferate, the need to erase specific concepts—such as copyrighted characters, artistic styles, or inappropriate content—has become critical. However, existing erasing methods often suffer from "concept erosion," where removing one target inadvertently degrades unrelated concepts, and a lack of transferability across different model versions. This paper introduces the Semi-Permeable Membrane (SPM), a lightweight 1D adapter that precisely erases target concepts while preserving the model's overall generative quality. SPMs are training-free transferable, allowing an adapter trained on one model to be instantly applied to others. Our results demonstrate that SPM achieves state-of-the-art precision in multi-concept erasing and significantly reduces harmful content generation with minimal parameter overhead.
Introduction
The rapid adoption of Diffusion Models (DMs) has raised significant concerns regarding copyright infringement, privacy, and the generation of harmful content. While researchers have proposed various "concept erasing" techniques, these methods typically face two major hurdles. First, concept erosion: fine-tuning a model to forget a specific concept often causes "parameter drift," leading to the degradation of unrelated images. This effect compounds as more concepts are erased, eventually causing the model's quality to collapse. Second, lack of transferability: current methods require expensive, model-specific fine-tuning. An erasure strategy for Stable Diffusion (SD) v1.4 cannot be easily applied to SD v1.5 or community-tuned variants without retraining.
To address these challenges, we present the Semi-Permeable Membrane (SPM). Inspired by biological membranes that selectively allow substances to pass, the SPM acts as a modular filter for diffusion models. It is a one-dimensional adapter that can be "plugged in" to erase specific concepts without invasive full-parameter updates.
The SPM Framework
The SPM framework is built on three core technical innovations designed for precision and modularity.
1. 1D Lightweight Adapter
Instead of fine-tuning the entire U-Net, we inject a rank-1 adapter into the diffusion layers. The structure follows $y = Wx + (v_{reg}^T x) \cdot v_{sig}$, where $v_{sig}$ represents the erasing signal and $v_{reg}$ is a trainable regulator. This approach adds only about 0.05% parameter overhead, making it extremely efficient.
2. Latent Anchoring (LA)
To combat concept erosion during the fine-tuning phase, we employ Latent Anchoring. We sample the latent space surrounding the target concept and "anchor" these neighboring representations to their original outputs. This forces the model to maintain its original performance on all semantic regions except for the narrow target being erased.
3. Facilitated Transport (FT)
At inference time, we use Facilitated Transport to dynamically manage the adapter's influence. By calculating CLIP similarity between the user's prompt and the erased concept, the system determines the SPM's activation level. If a prompt is unrelated to the target, the "membrane" remains closed, ensuring zero impact on the generated image.
Experimental Evaluation
We evaluated the SPM across diverse domains, including object removal (e.g., Snoopy, Mickey Mouse), artistic styles (e.g., Van Gogh), and safety benchmarks.
- Safety and Nudity Mitigation: Using the I2P (Inappropriate Image Prompts) benchmark, which contains over 4,700 risky prompts, we tested the SPM's ability to suppress NSFW content.
- General Quality: We used COCO-30k to measure the Fréchet Inception Distance (FID), ensuring that the overall generative capability of the base model remained intact after erasure.
- Multi-Concept Erasure: We tested the model's stability when progressively erasing up to 20 different targets simultaneously.
Key Results and Insights
The SPM framework demonstrates significant advantages over previous state-of-the-art (SOTA) methods like ESD and SA.
- Precision and Stability: In multi-concept erasing scenarios, SPM maintained a near-constant FID score. In contrast, previous methods saw their FID scores explode as more concepts were removed, indicating a total loss of image quality.
- Efficiency: By eliminating the need for repetitive training across different model versions, SPM achieves a 160x speedup in deployment for multi-model environments.
- Safety Performance: On the I2P benchmark, SPM reduced the generation of inappropriate content by 85.3% relative to the base SD v1.4 model.
- Universal Transferability: A key breakthrough is the ability to transfer SPMs without retraining. For example, a "Nudity SPM" trained on SD v1.4 was successfully applied to community models like ChillOutMix, effectively "clothing" generated characters in a model it had never seen during training.
Conclusion
The Semi-Permeable Membrane (SPM) represents a shift from "breaking" models to "filtering" them. By utilizing a 1D rank-1 adapter, we prove that minimal intervention is sufficient to steer massive diffusion models away from specific concepts. Its modularity and training-free transferability make it a practical and scalable solution for the evolving legal and ethical landscape of AI-generated content. Future work will focus on refining the boundaries between closely related semantic concepts to further enhance precision.
Summary updated