Introduction
Personalization techniques such as DreamBooth and Low-Rank Adaptation (LoRA) have revolutionized the way users interact with pre-trained diffusion models. These tools allow for the efficient "teaching" of specific subjects—like a personal pet—or unique artistic styles to a model. However, a significant hurdle remains: continual learning. When a model is taught a sequence of concepts (e.g., a "sneaker" in Task 1 followed by a "teapot" in Task 2), it often suffers from catastrophic forgetting, where the new knowledge overwrites or degrades the previously learned concepts.
This research explores methods to mitigate this forgetting under a strict continual learning regime. In this scenario, the system only has access to the current model and a single active adapter; no historical adapters or data from previous tasks are stored. The goal is to find an optimal way to update this single adapter so it can learn new concepts while preserving the integrity of old ones.
Methodology
The authors evaluate four primary strategies for updating the model across a sequence of 10 tasks. These methods vary in how they initialize new weights and how they merge them into the existing model structure.
- Naïve Continual Fine-Tuning: The baseline approach where the adapter from the previous task is simply fine-tuned further on the new task's data. This often leads to significant weight drift.
- Merge & Initialization: For each new task, the LoRA adapter weights are re-initialized. After training on the new task, these fresh weights are merged into the existing adapter. This approach encourages the model to find independent low-dimensional spaces for each concept.
- Merge & Orthogonal Initialization: This extends the previous method by using Singular Value Decomposition (SVD) to force the new adapter's weights to be mathematically orthogonal (perpendicular) to the previous weights, theoretically minimizing interference.
- Magnitude-based Merging (MAGMAX): A selection strategy that retains only the weights with the highest magnitudes during merging, based on the hypothesis that the strongest weights represent the most critical features.
Experimental Setup and Datasets
The methods were benchmarked on two distinct types of personalization tasks:
- Object Personalization (DreamBooth Dataset): Consisting of 30 different objects across 15 classes, such as specific dogs, cats, and household items.
- Style Personalization (UnlearnCanvas): This dataset involves rendering various objects in 60 different artistic styles, ranging from "Byzantine" to "Dadaism."
Key Insights and Results
The study revealed that the Merge & Initialization strategy provided the most effective balance between learning new tasks (plasticity) and retaining old ones (stability).
The Power of a Fresh Start
By resetting adapter weights for each task, the model avoids the cumulative "drift" that occurs when continuously nudging the same parameters. This creates a form of sparse adaptation that is significantly easier to merge without breaking previous knowledge. In object personalization, this method achieved the highest average scores (0.675 CLIP-I / 0.457 DINO), outperforming the Naïve approach.
Style vs. Substance
One of the most striking findings was that style personalization is considerably more difficult than object personalization. The authors hypothesize that "task vectors" for different artistic styles are more closely aligned in the model's latent space, making them more prone to interference. In style tasks, the Merge & Initialization method cut the forgetting rate nearly in half compared to the Naïve approach (0.131 vs 0.184 CSD).
The Limits of Orthogonality and Magnitude
While orthogonal initialization showed high plasticity, it suffered from poor stability, especially in style tasks. Similarly, while MAGMAX was effective at preventing forgetting, it acted as a "magnitude trap," severely limiting the model's ability to learn complex new traits and resulting in lower overall image quality.
Conclusion and Future Directions
While the proposed merging strategies significantly outperform standard fine-tuning, performance still gradually trends downward as the number of tasks increases. The strict memory constraints of this regime make it a challenging frontier. Future work may involve combining these merging techniques with advanced regularization methods like Elastic Weight Consolidation (EWC) or exploring dynamic rank adjustment to better accommodate tasks of varying complexity.
Summary updated