GRASP: Guided Residual Adapters with Sample-wise Partitioning

Felix Nützel, Mischa Dombrowski, Bernhard KainzarXiv 2025 · DOI · arXiv

Abstract

Diffusion models often struggle to generate high-quality, diverse images for rare "tail" classes, a phenomenon known as mode collapse. This paper introduces GRASP (Guided Residual Adapters with Sample-wise Partitioning), a framework that leverages external priors to partition data into semantically coherent clusters. By training specialized, parallel adapters for each cluster in a routing-free manner, GRASP effectively mitigates gradient conflicts during fine-tuning. Experimental results on medical and natural image datasets demonstrate that GRASP significantly improves the quality and diversity of generated images for long-tail distributions, outperforming existing mixture-of-experts and standard fine-tuning approaches.

Introduction

Modern text-to-image diffusion models have achieved remarkable success in generating high-fidelity images. However, their performance often degrades when dealing with long-tail distributions, where a few "head" classes dominate the training data while many "tail" classes are underrepresented. This is particularly critical in specialized domains like medical imaging, where "healthy" cases are abundant, but specific pathologies are rare.

The core issue identified by the authors is gradient conflict. During joint training on diverse classes, the strong gradient signals from frequent classes often overshadow or conflict with the signals from rare classes. This leads to mode collapse, where the model produces repetitive, low-diversity images for the tail classes. GRASP addresses this by isolating these conflicting signals through a structured partitioning and adapter-based approach.

Comparison of Normal Adapters vs. GRASP. Normal adapters suffer from large gradient conflicts leading to low image diversity, while GRASP partitions data to minimize conflicts and improve diversity.

Methodology: Guided Residual Adapters

GRASP introduces a lightweight fine-tuning framework designed to minimize gradient clashes without the instability of learned routing.

Sample-wise Partitioning

Instead of relying on a learned gating network, which can be unstable and prone to "expert collapse," GRASP uses a static external prior. Training samples are partitioned into semantically coherent clusters (e.g., Head, Medium, Tail, or Healthy vs. Pathological) based on class labels or text embeddings. This ensures that samples within each partition have aligned gradients, reducing internal conflict. The authors demonstrate that semantic similarity serves as an excellent proxy for gradient alignment.

Guided Residual Adapters

The architecture injects cluster-specific residual adapters into the feedforward layers of a frozen, pre-trained diffusion transformer (specifically the Flux architecture). These adapters are trained to specialize in the specific nuances of their assigned partition. The authors found that using "wider" adapters (higher dimensionality) in later layers was more effective than thinner adapters distributed throughout the network.

Routing-Free Architecture

Unlike standard Mixture-of-Experts (MoE) models, GRASP deterministically routes samples to their designated adapters based on the pre-defined partitions. This eliminates the need for a complex gating mechanism and ensures that every expert specializes as intended.

Overview of the GRASP framework: (a) Sample-wise partitioning based on gradient directions/semantic similarity, and (b) the Guided Residual Adapter architecture integrated with a frozen base layer.

Evaluation and Results

The researchers evaluated GRASP across three benchmarks: MIMIC-CXR-LT, NIH-CXR-LT, and ImageNet-LT.

Key Performance Metrics

  • Image Quality: On the MIMIC-CXR-LT dataset, GRASP achieved an 80% reduction in FID compared to standard fine-tuning, indicating significantly higher realism.
  • Diversity: Tail-class diversity, measured by Coverage, increased by up to 44%, effectively mitigating mode collapse.
  • Downstream Utility: Synthetic images generated by GRASP were used to train a disease classifier, resulting in a 6 percentage point improvement in F1-score for rare labels. This confirms that the generated images are medically meaningful and useful for data augmentation.

Insights

A key finding was the importance of the "Healthy" Expert. In medical imaging, dedicating a specific expert to normal cases significantly improved the performance of experts handling rare pathologies. Furthermore, the method's success on ImageNet-LT demonstrates its generalizability beyond the medical domain.

Discussion and Future Work

GRASP provides a robust and simple solution to the mode collapse problem in long-tail diffusion training. By replacing learned routing with static, guided partitioning, it achieves superior stability and performance.

Despite these gains, the "extreme tail"—classes with very few examples—remains a challenge. Future research will explore hierarchical partitioning strategies and the development of universal experts that can generalize across datasets. Additionally, integrating GRASP with dynamic routing or reinforcement learning could further refine expert utilization during the generation process.

Summary updated

BibTeX
@misc{nutzel2025grasp,
  author = {Felix Nützel and Mischa Dombrowski and Bernhard Kainz},
  title = {{GRASP: Guided Residual Adapters with Sample-wise Partitioning}},
  year = {2025},
  eprint = {2512.01675},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2512.01675},
}
Figure