Introduction
Generating high-quality medical images, such as X-rays, is a significant challenge in healthcare AI due to the scarcity of annotated data. While Latent Diffusion Models (LDMs) like Stable Diffusion have shown remarkable generative capabilities, fine-tuning these large-scale models on specialized medical datasets is computationally prohibitive.
A common solution is Low-Rank Adaptation (LoRA), which introduces small, trainable rank-decomposition matrices into the frozen pre-trained model. However, standard LoRA employs a uniform rank across all layers, which is inefficient as different layers contribute varyingly to task-specific adaptation. Existing adaptive methods often start with a high rank and prune down, leading to high initial memory costs and the need for a predefined upper bound. SeLoRA (Self-Expanding Low-Rank Adaptation) addresses these issues by dynamically growing the rank of adapter layers from the bottom up.
Methodology: The Self-Expanding Mechanism
SeLoRA starts with a minimal rank ($r=1$) for every layer and expands capacity only where it is most beneficial. This process is governed by three main components:
- Dynamic Expansion: The model periodically evaluates (every $t$ training steps) whether to increase the rank of a specific layer.
- Fisher Information (FI) Guidance: To identify layers requiring more capacity, SeLoRA calculates an "FI-Ratio." Fisher Information quantifies how sensitive the model's output is to specific weights. If the ratio between the information of current weights and potential expanded weights exceeds a threshold ($\lambda$), the layer's rank is incremented.
- Zero-Perturbation Initialization: To maintain training stability and prevent "forgetting," new parameters are initialized such that the model's output remains unchanged immediately after expansion. Specifically, matrix $B$ is initialized to zero, while matrix $A$ uses Kaiming uniform initialization.
Experimental Setup
The effectiveness of SeLoRA was evaluated on two chest X-ray benchmarks:
- IU X-RAY: A dataset containing 3,955 radiology reports paired with frontal projection images.
- Montgomery County CXR: A compact dataset of 138 frontal X-rays, used to test the model's performance under extreme data scarcity.
Results and Discussion
Strategic Parameter Allocation
A key advantage of SeLoRA is its ability to intelligently allocate parameters. Analysis of the rank distribution reveals that the model naturally assigns higher ranks to cross-attention layers. These layers are critical as they manage the interaction between text prompts (e.g., radiology reports) and image features.
Performance and Efficiency
SeLoRA consistently outperformed standard LoRA and other adaptive methods like AdaLoRA and DyLoRA:
- Image Fidelity: On the Montgomery dataset, SeLoRA achieved a Fréchet Inception Distance (FID) of 205.54, a significant improvement over standard LoRA's 461.80.
- Prompt Alignment: The model demonstrated superior CLIP scores, indicating better alignment between generated images and their textual descriptions.
- Efficiency: On the IU-XRay dataset, SeLoRA achieved state-of-the-art results using approximately half the trainable parameters of a fixed-rank LoRA (rank 4) for the U-Net and text encoder.
Pathology Preservation
In medical contexts, the preservation of subtle pathological features is paramount. SeLoRA proved more adept at capturing specific clinical details, such as nodules or lesions, compared to standard LoRA, which often produced visually "cleaner" but less clinically accurate images.
Conclusion and Future Work
SeLoRA offers a parameter-efficient and flexible approach to fine-tuning LDMs for medical imaging. By starting small and expanding based on information theory, it avoids the overhead of pruning-based methods.
Future research will focus on:
- Hyperparameter Optimization: Investigating the sensitivity of the expansion threshold $\lambda$ and step interval $t$ across different medical modalities.
- 3D Synthesis: Extending the framework to 3D data like CT or MRI scans, where data scarcity and computational demands are even more acute.
- Computational Cost: Refining the Fisher Information calculation to further minimize training overhead.
Summary updated