Time-Varying LoRA: Towards Effective Cross-Domain Fine-Tuning of Diffusion Models

Zhan Zhuang, Yulong Zhang, Xuehao Wang, Jiangang Lu, Ying WeiNeurIPS 2024 · OpenReview

TL;DR

Terra (Time-varying low-rank adapter) is a novel fine-tuning framework that introduces a continuous time variable into the standard LoRA architecture. By constructing a parameter manifold, Terra enables smooth, high-fidelity transitions between image domains. It serves as a powerful, parameter-efficient "plugin" for solving domain adaptation and generalization challenges in diffusion models.

The Challenge: Static Adapters in a Dynamic World

While large-scale diffusion models like Stable Diffusion XL are incredible at generating high-fidelity images, they often struggle with "domain flow"—the gradual, continuous transition between different styles, subjects, or environments.

Standard fine-tuning techniques like LoRA (Low-Rank Adaptation) are highly efficient but inherently static; they typically specialize a model for one specific domain at a time. If a user wants to bridge the gap between a "source" domain (e.g., sketches) and a "target" domain (e.g., photorealism), they usually have to train separate LoRAs and find complex, often clumsy ways to interpolate between them. This approach is parameter-inefficient and frequently results in "ghosting" or unnatural artifacts during transitions. The authors of this paper aim to solve this by creating a single, unified adapter capable of navigating the entire space between domains smoothly.

Introducing Terra: Time-Varying LoRA

The core innovation of this paper is Terra, which fundamentally modifies the standard LoRA weight update mechanism. In a typical LoRA, the weight change is represented as $\Delta W = W_{up} W_{down}$. Terra introduces a time-varying middle matrix $K(t)$ controlled by a continuous time variable $t \in [0, 1]$:

$$\Delta W(t) = W_{up} K(t) W_{down}$$

Here, $K(t)$ is a differentiable function of a learned matrix $W_{mid}$ and the time variable $t$. The authors explore several mathematical forms for this function, including:

  • Linear: $K(t) = tW_{mid} + I$ (where $I$ is the identity matrix).
  • Exponential: $K(t) = \exp(tW_{mid})$.
  • Cosine-Sine: Using trigonometric functions to capture more complex, non-linear relationships.
From static LoRA to the time-varying Terra manifold

By setting $t=0$ for the source domain and $t=1$ for the target domain, Terra learns a continuous "parameter manifold" that seamlessly connects the two. This allows the diffusion model to generate images at any intermediate point $t$, effectively "morphing" content or style without requiring multiple independent adapters.

Key Insights and Applications

Beyond simple image morphing, Terra's continuous manifold unlocks generation-based approaches for complex machine learning tasks:

  • Theoretical Expressive Power: The authors prove mathematically that a single Terra adapter can effectively implement the functionality of two separate LoRAs while using significantly fewer parameters, making it highly efficient.
  • Unsupervised Domain Adaptation (UDA): For UDA, Terra is used to generate a "transitive" source domain. By transforming source images into the target style while preserving their original labels, Terra creates a high-quality augmented dataset. This bridges the visual gap and makes the final adaptation task much easier for standard UDA methods.
  • Domain Generalization (DG): In DG, the goal is to perform well on unseen domains. Terra achieves this by using a "$t$-predictor" network to map different source domains onto the manifold. By sampling random $t$ values, the framework generates an "expanded" set of domains with diverse, interpolated styles, forcing the model to learn more robust, generalized features.
Terra integrated into Domain Generalization pipelines

Experimental Results

The authors evaluated Terra across Generative Interpolation, UDA, and DG tasks using benchmark datasets like Office-Home, VisDA, and PACS.

Terra demonstrated superior performance in both qualitative and quantitative metrics:

  • Image Morphing: Terra achieved significantly lower Frechet Inception Distance (FID) and Perceptual Path Length (PPL) compared to standard LoRA interpolation and specialized tools like DiffMorpher, indicating smoother and more realistic visual transitions.
  • UDA Performance: When integrated as a plugin for existing UDA methods (like ELS), Terra boosted average accuracy on the Office-Home dataset by 4.42% and on VisDA by 3.46%, reaching state-of-the-art levels.
  • DG Performance: On the PACS dataset, combining Terra with the SWAD method improved accuracy from 88.27% to 89.34%, outperforming other diffusion-based augmentation methods like DomainDiff.

Limitations and Future Work

While highly effective, Terra does present a few challenges. Generating large-scale augmented datasets for UDA and DG requires significant additional storage space. Furthermore, the fine-tuned model can still be heavily influenced by the "prior" knowledge of the base foundation model (e.g., SDXL), which might occasionally conflict with specific downstream task requirements. Future work aims to extend Terra to handle more complex scenarios, such as multi-modal inputs (image, text, and audio) and even larger, more diverse datasets.

Summary updated

Abstract

Large-scale diffusion models are adept at generating high-fidelity images and facilitating image editing and interpolation. However, they have limitations when tasked with generating images in dynamic, evolving domains. In this paper, we introduce Terra, a novel Time-varying low-rank adapter that offers a fine-tuning framework specifically tailored for domain flow generation. The key innovation of Terra lies in its construction of a continuous parameter manifold through a time variable, with its expressive power analyzed theoretically. This framework not only enables interpolation of image content and style but also offers a generation-based approach to address the domain shift problems in unsupervised domain adaptation and domain generalization. Specifically, Terra transforms images from the source domain to the target domain and generates interpolated domains with various styles to bridge the gap between domains and enhance the model generalization, respectively. We conduct extensive experiments on various benchmark datasets, empirically demonstrate the effectiveness of Terra. Our source code is publicly available on https://github.com/zwebzone/terra.
BibTeX
@inproceedings{zhuang2024time,
  author = {Zhan Zhuang and Yulong Zhang and Xuehao Wang and Jiangang Lu and Ying Wei and Yu Zhang},
  title = {{Time-Varying LoRA: Towards Effective Cross-Domain Fine-Tuning of Diffusion Models}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2024},
  url = {https://openreview.net/forum?id=SgODU2mx9T},
}
Figure