One-dimensional Adapter to Rule Them All: Concepts, Diffusion Models and Erasing Applications

Mengyao Lyu, Yuhong Yang, Haiwen Hong, Hui Chen, Xuan Jin, Yuan He, Hui Xue, Jungong Han, Guiguang Ding

Abstract

As text-to-image diffusion models (DMs) proliferate, the need to erase specific concepts—such as copyrighted characters, artistic styles, or inappropriate content—has become critical. However, existing erasing methods often suffer from "concept erosion," where removing one target inadvertently degrades unrelated concepts, and a lack of transferability across different model versions. This paper introduces the Semi-Permeable Membrane (SPM), a lightweight 1D adapter that precisely erases target concepts while preserving the model's overall generative quality. SPMs are training-free transferable, allowing an adapter trained on one model to be instantly applied to others. Our results demonstrate that SPM achieves state-of-the-art precision in multi-concept erasing and significantly reduces harmful content generation with minimal parameter overhead.

Introduction

The rapid adoption of Diffusion Models (DMs) has raised significant concerns regarding copyright infringement, privacy, and the generation of harmful content. While researchers have proposed various "concept erasing" techniques, these methods typically face two major hurdles. First, concept erosion: fine-tuning a model to forget a specific concept often causes "parameter drift," leading to the degradation of unrelated images. This effect compounds as more concepts are erased, eventually causing the model's quality to collapse. Second, lack of transferability: current methods require expensive, model-specific fine-tuning. An erasure strategy for Stable Diffusion (SD) v1.4 cannot be easily applied to SD v1.5 or community-tuned variants without retraining.

To address these challenges, we present the Semi-Permeable Membrane (SPM). Inspired by biological membranes that selectively allow substances to pass, the SPM acts as a modular filter for diffusion models. It is a one-dimensional adapter that can be "plugged in" to erase specific concepts without invasive full-parameter updates.

Comparison of SPM against previous methods. SPM prevents concept erosion and enables training-free transfer to other models.

The SPM Framework

The SPM framework is built on three core technical innovations designed for precision and modularity.

1. 1D Lightweight Adapter

Instead of fine-tuning the entire U-Net, we inject a rank-1 adapter into the diffusion layers. The structure follows $y = Wx + (v_{reg}^T x) \cdot v_{sig}$, where $v_{sig}$ represents the erasing signal and $v_{reg}$ is a trainable regulator. This approach adds only about 0.05% parameter overhead, making it extremely efficient.

2. Latent Anchoring (LA)

To combat concept erosion during the fine-tuning phase, we employ Latent Anchoring. We sample the latent space surrounding the target concept and "anchor" these neighboring representations to their original outputs. This forces the model to maintain its original performance on all semantic regions except for the narrow target being erased.

3. Facilitated Transport (FT)

At inference time, we use Facilitated Transport to dynamically manage the adapter's influence. By calculating CLIP similarity between the user's prompt and the erased concept, the system determines the SPM's activation level. If a prompt is unrelated to the target, the "membrane" remains closed, ensuring zero impact on the generated image.

The SPM architecture featuring Latent Anchoring during fine-tuning and Facilitated Transport during generation.

Experimental Evaluation

We evaluated the SPM across diverse domains, including object removal (e.g., Snoopy, Mickey Mouse), artistic styles (e.g., Van Gogh), and safety benchmarks.

  • Safety and Nudity Mitigation: Using the I2P (Inappropriate Image Prompts) benchmark, which contains over 4,700 risky prompts, we tested the SPM's ability to suppress NSFW content.
  • General Quality: We used COCO-30k to measure the Fréchet Inception Distance (FID), ensuring that the overall generative capability of the base model remained intact after erasure.
  • Multi-Concept Erasure: We tested the model's stability when progressively erasing up to 20 different targets simultaneously.

Key Results and Insights

The SPM framework demonstrates significant advantages over previous state-of-the-art (SOTA) methods like ESD and SA.

  • Precision and Stability: In multi-concept erasing scenarios, SPM maintained a near-constant FID score. In contrast, previous methods saw their FID scores explode as more concepts were removed, indicating a total loss of image quality.
  • Efficiency: By eliminating the need for repetitive training across different model versions, SPM achieves a 160x speedup in deployment for multi-model environments.
  • Safety Performance: On the I2P benchmark, SPM reduced the generation of inappropriate content by 85.3% relative to the base SD v1.4 model.
  • Universal Transferability: A key breakthrough is the ability to transfer SPMs without retraining. For example, a "Nudity SPM" trained on SD v1.4 was successfully applied to community models like ChillOutMix, effectively "clothing" generated characters in a model it had never seen during training.

Conclusion

The Semi-Permeable Membrane (SPM) represents a shift from "breaking" models to "filtering" them. By utilizing a 1D rank-1 adapter, we prove that minimal intervention is sufficient to steer massive diffusion models away from specific concepts. Its modularity and training-free transferability make it a practical and scalable solution for the evolving legal and ethical landscape of AI-generated content. Future work will focus on refining the boundaries between closely related semantic concepts to further enhance precision.

Summary updated

Abstract

The prevalent use of commercial and open-source diffusion models (DMs) for text-to-image generation prompts risk mitigation to prevent undesired behaviors. Existing concept erasing methods in academia are all based on full parameter or specification-based fine-tuning, from which we observe the following issues: 1) Generation alternation towards erosion: Parameter drift during target elimination causes alternations and potential deformations across all generations, even eroding other concepts at varying degrees, which is more evident with multi-concept erased; 2) Transfer inability & deployment inefficiency: Previous model-specific erasure impedes the flexible combination of concepts and the training-free transfer towards other models, resulting in linear cost growth as the deployment scenarios increase. To achieve non-invasive, precise, customizable, and transferable elimination, we ground our erasing framework on one-dimensional adapters to erase multiple concepts from most DMs at once across versatile erasing applications. The concept-SemiPermeable structure is injected as a Membrane (SPM) into any DM to learn targeted erasing, and meantime the alteration and erosion phenomenon is effectively mitigated via a novel Latent Anchoring fine-tuning strategy. Once obtained, SPMs can be flexibly combined and plug-and-play for other DMs without specific re-tuning, enabling timely and efficient adaptation to diverse scenarios. During generation, our Facilitated Transport mechanism dynamically regulates the permeability of each SPM to respond to different input prompts, further minimizing the impact on other concepts. Quantitative and qualitative results across ~40 concepts, 7 DMs and 4 erasing applications have demonstrated the superior erasing of SPM. Our code and pre-tuned SPMs are available on the project page https://lyumengyao.github.io/projects/spm.
BibTeX
@inproceedings{lyu2023one,
  author = {Mengyao Lyu and Yuhong Yang and Haiwen Hong and Hui Chen and Xuan Jin and Yuan He and Hui Xue and Jungong Han and Guiguang Ding},
  title = {{One-dimensional Adapter to Rule Them All: Concepts, Diffusion Models and Erasing Applications}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2023},
}
Figure