Low-Rank Continual Personalization of Diffusion Models

Lukasz Staniszewski, Katarzyna Zaleska, Kamil Deja

Introduction

Personalization techniques such as DreamBooth and Low-Rank Adaptation (LoRA) have revolutionized the way users interact with pre-trained diffusion models. These tools allow for the efficient "teaching" of specific subjects—like a personal pet—or unique artistic styles to a model. However, a significant hurdle remains: continual learning. When a model is taught a sequence of concepts (e.g., a "sneaker" in Task 1 followed by a "teapot" in Task 2), it often suffers from catastrophic forgetting, where the new knowledge overwrites or degrades the previously learned concepts.

This research explores methods to mitigate this forgetting under a strict continual learning regime. In this scenario, the system only has access to the current model and a single active adapter; no historical adapters or data from previous tasks are stored. The goal is to find an optimal way to update this single adapter so it can learn new concepts while preserving the integrity of old ones.

Methodology

The authors evaluate four primary strategies for updating the model across a sequence of 10 tasks. These methods vary in how they initialize new weights and how they merge them into the existing model structure.

Overview of the four investigated strategies for continual personalization of diffusion models, ranging from naive fine-tuning to magnitude-based selection.
  1. Naïve Continual Fine-Tuning: The baseline approach where the adapter from the previous task is simply fine-tuned further on the new task's data. This often leads to significant weight drift.
  2. Merge & Initialization: For each new task, the LoRA adapter weights are re-initialized. After training on the new task, these fresh weights are merged into the existing adapter. This approach encourages the model to find independent low-dimensional spaces for each concept.
  3. Merge & Orthogonal Initialization: This extends the previous method by using Singular Value Decomposition (SVD) to force the new adapter's weights to be mathematically orthogonal (perpendicular) to the previous weights, theoretically minimizing interference.
  4. Magnitude-based Merging (MAGMAX): A selection strategy that retains only the weights with the highest magnitudes during merging, based on the hypothesis that the strongest weights represent the most critical features.

Experimental Setup and Datasets

The methods were benchmarked on two distinct types of personalization tasks:

  • Object Personalization (DreamBooth Dataset): Consisting of 30 different objects across 15 classes, such as specific dogs, cats, and household items.
  • Style Personalization (UnlearnCanvas): This dataset involves rendering various objects in 60 different artistic styles, ranging from "Byzantine" to "Dadaism."

Key Insights and Results

The study revealed that the Merge & Initialization strategy provided the most effective balance between learning new tasks (plasticity) and retaining old ones (stability).

Detailed view of the Merge & Initialization strategy, which resets weights for each task to prevent cumulative drift.

The Power of a Fresh Start

By resetting adapter weights for each task, the model avoids the cumulative "drift" that occurs when continuously nudging the same parameters. This creates a form of sparse adaptation that is significantly easier to merge without breaking previous knowledge. In object personalization, this method achieved the highest average scores (0.675 CLIP-I / 0.457 DINO), outperforming the Naïve approach.

Style vs. Substance

One of the most striking findings was that style personalization is considerably more difficult than object personalization. The authors hypothesize that "task vectors" for different artistic styles are more closely aligned in the model's latent space, making them more prone to interference. In style tasks, the Merge & Initialization method cut the forgetting rate nearly in half compared to the Naïve approach (0.131 vs 0.184 CSD).

The Limits of Orthogonality and Magnitude

While orthogonal initialization showed high plasticity, it suffered from poor stability, especially in style tasks. Similarly, while MAGMAX was effective at preventing forgetting, it acted as a "magnitude trap," severely limiting the model's ability to learn complex new traits and resulting in lower overall image quality.

Conclusion and Future Directions

While the proposed merging strategies significantly outperform standard fine-tuning, performance still gradually trends downward as the number of tasks increases. The strict memory constraints of this regime make it a challenging frontier. Future work may involve combining these merging techniques with advanced regularization methods like Elastic Weight Consolidation (EWC) or exploring dynamic rank adjustment to better accommodate tasks of varying complexity.

Summary updated

Abstract

Recent personalization methods for diffusion models, such as Dreambooth and LoRA, allow fine-tuning pre-trained models to generate new concepts. However, applying these techniques across consecutive tasks in order to include, e.g., new objects or styles, leads to a forgetting of previous knowledge due to mutual interference between their adapters. In this work, we tackle the problem of continual customization under a rigorous regime with no access to past tasks' adapters. In such a scenario, we investigate how different adapters' initialization and merging methods can improve the quality of the final model. To that end, we evaluate the naive continual fine-tuning of customized models and compare this approach with three methods for consecutive adapters' training: sequentially merging new adapters, merging orthogonally initialized adapters, and updating only relevant task-specific weights. In our experiments, we show that the proposed techniques mitigate forgetting when compared to the naive approach. In our studies, we show different traits of selected techniques and their effect on the plasticity and stability of the continually adapted model. Repository with the code is available at https://github.com/luk-st/continual-lora.
BibTeX
@inproceedings{staniszewski2025low,
  author = {Lukasz Staniszewski and Katarzyna Zaleska and Kamil Deja},
  title = {{Low-Rank Continual Personalization of Diffusion Models}},
  booktitle = {SCOPE @ ICLR},
  year = {2025},
  url = {https://openreview.net/forum?id=XsXhhq4Kvx},
}
Figure