Introduction
Large-scale diffusion models have revolutionized generative AI but remain notoriously data-hungry. Adapting these models to niche target populations often requires significant data, which is frequently unavailable. While "checkpoint merging"—averaging the weights of pre-trained models—is a common heuristic, it often lacks theoretical grounding and produces bimodal artifacts rather than smooth semantic blends. This paper introduces ScoreFusion, a mathematically principled framework for model blending based on Kullback–Leibler (KL) barycenters.
The ScoreFusion Methodology
ScoreFusion shifts the interpolation from the parameter space (weights) to the function space (scores). The approach is rooted in the concept of the KL barycenter: a distribution that minimizes the weighted sum of KL divergences to a set of auxiliary distributions.
Linearity of Scores
The core theoretical insight is that the score function of a KL barycenter is a linear combination of the score functions of the auxiliary models. This allows for a two-phase process:
- Calibration: Using a small target dataset (as few as 64 samples), the optimal weights ((\lambda)) for the auxiliary scores are determined via a simple linear regression task.
- Sampling: New images are generated by following a diffusion path where the score at each step is the weighted sum of the auxiliary models' scores.
Experimental Evaluation
The authors validated ScoreFusion across diverse tasks, including EMNIST digit matching and high-resolution portrait generation using SDXL 1.0. In the SDXL experiments, ScoreFusion was used to fuse specialized LoRA adapters, such as those representing distinct demographic profiles.
Unlike weight averaging, which often results in jarring identity shifts or bimodal results, ScoreFusion produced smooth transitions and explored the "middle ground" of the distribution, generating diverse faces that were rare in the original source models.
Key Findings
- Sample Efficiency: ScoreFusion demonstrates a "dimension-free" sample complexity, meaning the number of samples required to find optimal fusion weights does not scale with image resolution.
- Superior Interpolation: Quantitative analysis using CLIP embeddings showed that ScoreFusion captures the target distribution more accurately than fine-tuning or checkpoint merging, especially in low-data regimes.
- Robustness: Synthetic tests confirmed that ScoreFusion consistently outperforms baselines in capturing complex multi-modal distributions until target datasets become sufficiently large.
Conclusion
ScoreFusion provides a robust, theoretically sound alternative to ad-hoc model merging. By leveraging the linearity of scores in KL barycenters, it enables the creation of high-quality generative models for specialized populations with minimal data. Future work may extend this "gradient flow" approach to other domains such as natural language processing and reinforcement learning.
Summary updated