SeLoRA: Self-Expanding Low-Rank Adaptation of Latent Diffusion Model for Medical Image Synthesis

Guang Yang, Ieee Senior Member, Yuchen Mao, Hongwei Li, Wei Pang, Giorgos Papanastasiou, Guang Yang, Chengjia Wang

Introduction

Generating high-quality medical images, such as X-rays, is a significant challenge in healthcare AI due to the scarcity of annotated data. While Latent Diffusion Models (LDMs) like Stable Diffusion have shown remarkable generative capabilities, fine-tuning these large-scale models on specialized medical datasets is computationally prohibitive.

A common solution is Low-Rank Adaptation (LoRA), which introduces small, trainable rank-decomposition matrices into the frozen pre-trained model. However, standard LoRA employs a uniform rank across all layers, which is inefficient as different layers contribute varyingly to task-specific adaptation. Existing adaptive methods often start with a high rank and prune down, leading to high initial memory costs and the need for a predefined upper bound. SeLoRA (Self-Expanding Low-Rank Adaptation) addresses these issues by dynamically growing the rank of adapter layers from the bottom up.

Methodology: The Self-Expanding Mechanism

SeLoRA starts with a minimal rank ($r=1$) for every layer and expands capacity only where it is most beneficial. This process is governed by three main components:

  1. Dynamic Expansion: The model periodically evaluates (every $t$ training steps) whether to increase the rank of a specific layer.
  2. Fisher Information (FI) Guidance: To identify layers requiring more capacity, SeLoRA calculates an "FI-Ratio." Fisher Information quantifies how sensitive the model's output is to specific weights. If the ratio between the information of current weights and potential expanded weights exceeds a threshold ($\lambda$), the layer's rank is incremented.
  3. Zero-Perturbation Initialization: To maintain training stability and prevent "forgetting," new parameters are initialized such that the model's output remains unchanged immediately after expansion. Specifically, matrix $B$ is initialized to zero, while matrix $A$ uses Kaiming uniform initialization.
The SeLoRA workflow: from initialization with minimal rank to progressive testing and conditional expansion based on Fisher Information.

Experimental Setup

The effectiveness of SeLoRA was evaluated on two chest X-ray benchmarks:

  • IU X-RAY: A dataset containing 3,955 radiology reports paired with frontal projection images.
  • Montgomery County CXR: A compact dataset of 138 frontal X-rays, used to test the model's performance under extreme data scarcity.

Results and Discussion

Strategic Parameter Allocation

A key advantage of SeLoRA is its ability to intelligently allocate parameters. Analysis of the rank distribution reveals that the model naturally assigns higher ranks to cross-attention layers. These layers are critical as they manage the interaction between text prompts (e.g., radiology reports) and image features.

Heatmap showing the dynamic rank allocation across different attention layers and weights. Higher ranks are concentrated in layers where complex text-to-image translation occurs.

Performance and Efficiency

SeLoRA consistently outperformed standard LoRA and other adaptive methods like AdaLoRA and DyLoRA:

  • Image Fidelity: On the Montgomery dataset, SeLoRA achieved a Fréchet Inception Distance (FID) of 205.54, a significant improvement over standard LoRA's 461.80.
  • Prompt Alignment: The model demonstrated superior CLIP scores, indicating better alignment between generated images and their textual descriptions.
  • Efficiency: On the IU-XRay dataset, SeLoRA achieved state-of-the-art results using approximately half the trainable parameters of a fixed-rank LoRA (rank 4) for the U-Net and text encoder.

Pathology Preservation

In medical contexts, the preservation of subtle pathological features is paramount. SeLoRA proved more adept at capturing specific clinical details, such as nodules or lesions, compared to standard LoRA, which often produced visually "cleaner" but less clinically accurate images.

Conclusion and Future Work

SeLoRA offers a parameter-efficient and flexible approach to fine-tuning LDMs for medical imaging. By starting small and expanding based on information theory, it avoids the overhead of pruning-based methods.

Future research will focus on:

  • Hyperparameter Optimization: Investigating the sensitivity of the expansion threshold $\lambda$ and step interval $t$ across different medical modalities.
  • 3D Synthesis: Extending the framework to 3D data like CT or MRI scans, where data scarcity and computational demands are even more acute.
  • Computational Cost: Refining the Fisher Information calculation to further minimize training overhead.

Summary updated

Abstract

The persistent challenge of medical image synthesis posed by the scarcity of annotated data and the need to synthesize "missing modalities" for multi-modal analysis, underscored the imperative development of effective synthesis methods. Recently, the combination of Low-Rank Adaptation (*LoRA*) with latent diffusion models (LDMs) has emerged as a viable approach for efficiently adapting pre-trained large language models, in the medical field. However, the direct application of *LoRA* assumes uniform ranking across all linear layers, overlooking the significance of different weight matrices, and leading to sub-optimal outcomes. Prior works on *LoRA* prioritize the reduction of trainable parameters, and there exists an opportunity to further tailor this adaptation process to the intricate demands of medical image synthesis. In response, we present *SeLoRA*, a Self-Expanding Low-Rank Adaptation module, that dynamically expands its ranking across layers during training, strategically placing additional ranks on crucial layers, to allow the model to elevate synthesis quality where it matters most. Our analysis shows that *SeLoRA* strikes the best balance between synthesis quality and training efficiency. The proposed method not only enables LDMs to fine-tune on medical data efficiently but also empowers the model to achieve improved image quality with minimal ranking. The code of our *SeLoRA* method is publicly available at https://anonymous.4open.science/r/SeLoRA-980D.
BibTeX
@misc{member2024selora,
  author = {Guang Yang, IEEE Senior Member and Yuchen Mao and Hongwei Li and Wei Pang and Giorgos Papanastasiou and Guang Yang and Chengjia Wang},
  title = {{SeLoRA: Self-Expanding Low-Rank Adaptation of Latent Diffusion Model for Medical Image Synthesis}},
  year = {2024},
  eprint = {2408.07196},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2408.07196},
}
Figure