ScoreFusion: Fusing Score-based Generative Models via Kullback–Leibler Barycenters

Hao Liu, Tony Junze Ye, Jose Blanchet, Nian Si

Abstract

We introduce ScoreFusion, a theoretically grounded method for fusing multiple pre-trained diffusion models that are assumed to generate from auxiliary populations. ScoreFusion is particularly useful for enhancing the generative modeling of a target population with limited observed data. Our starting point considers the family of KL barycenters of the auxiliary populations, which is proven to be an optimal parametric class in the KL sense, but difficult to learn. Nevertheless, by recasting the learning problem as score matching in denoising diffusion, we obtain a tractable way of computing the optimal KL barycenter weights. We prove a dimension-free sample complexity bound in total variation distance, provided that the auxiliary models are well-fitted for their own task and the auxiliary tasks combined capture the target well. The sample efficiency of ScoreFusion is demonstrated by learning handwritten digits. We also provide a simple adaptation of a Stable Diffusion denoising pipeline that enables sampling from the KL barycenter of two auxiliary checkpoints; on a portrait generation task, our method produces faces that enhance population heterogeneity relative to the auxiliary distributions.

Introduction

Diffusion models have become the standard for high-fidelity image generation, yet their reliance on massive datasets remains a significant hurdle for specialized or "long-tail" applications. When data for a target population is scarce, researchers typically turn to fine-tuning or checkpoint merging. However, fine-tuning on small datasets often leads to overfitting and a loss of diversity, while checkpoint merging lacks a rigorous theoretical foundation and frequently results in "identity jumps"—where the model toggles between discrete modes rather than producing a smooth blend.

We present ScoreFusion, a method that addresses these limitations by leveraging the mathematics of Kullback–Leibler (KL) barycenters. ScoreFusion allows for the principled combination of multiple pre-trained "expert" models to represent a new target population with extreme sample efficiency.

Methodology

The KL Barycenter Principle

A KL barycenter is a distribution that serves as a weighted average of several reference distributions. In the context of generative modeling, we assume the target population can be approximated by a barycenter of existing auxiliary models.

Linearity of Scores

The key mathematical breakthrough of ScoreFusion is the proof that the optimal score function (the gradient of the log-density) for a KL barycenter is a linear combination of the score functions of the auxiliary models. This observation transforms a complex non-parametric learning problem into a simple parametric one: finding the optimal weights ($\lambda$) for the linear combination.

The ScoreFusion Algorithm

  1. Calibration Phase: Using a small target dataset (as few as 64 samples), the algorithm learns the optimal weights for each pre-trained model by minimizing a score-matching loss. This optimization is convex and computationally efficient.
  2. Sampling Phase: During image generation, the model performs a standard denoising process. At each step, the noise prediction is calculated as the weighted sum of the outputs from the pre-trained auxiliary models.

Experimental Results

Sample Efficiency on EMNIST

On the Extended MNIST dataset, ScoreFusion was tasked with learning specific digit proportions (e.g., a 60/40 split of 7s and 9s). With only 64 samples, ScoreFusion achieved lower Negative Log-Likelihood (NLL) and more accurate proportion capture than both training from scratch and standard fine-tuning.

Semantic Blending in SDXL

We applied ScoreFusion to Stable Diffusion XL (SDXL) using two specialized checkpoints: one for "White male" portraits and one for "East Asian female" portraits.

Sample portrait generated using ScoreFusion, demonstrating high-fidelity facial features and the ability to capture specific demographic attributes.

Unlike traditional weight merging, which often produces abrupt shifts in identity, ScoreFusion enabled a smooth, continuous spectrum of physical appearances. The method produced portraits with enhanced population heterogeneity, including more neutral gender expressions and ambiguous ethnicities, effectively "filling the gaps" left by biased auxiliary models.

Portrait demonstrating the method's capacity for demographic blending and generating diverse facial structures.

Synthetic Mode Capture

In tests with bimodal Gaussian distributions, ScoreFusion accurately identified and captured both modes with minimal data. In contrast, standard diffusion models tended to produce a "smeared" distribution that failed to resolve the distinct peaks of the target population.

Key Insights

  • Dimension-Free Complexity: The authors prove a sample complexity bound that does not scale with the dimensionality of the data, provided the auxiliary models are well-fitted. This explains the method's success in high-resolution image generation with very few samples.
  • True Interpolation vs. Mixture: While weight merging often creates a bimodal mixture (e.g., generating either a cat or a dog), ScoreFusion creates a true barycenter, exploring the shared semantic space between distributions (e.g., a hybrid "cat-dog").

Limitations and Future Work

The success of ScoreFusion is dependent on the quality and "span" of the auxiliary models; it cannot generate concepts entirely absent from the reference set. Additionally, the calibration parameter $\tilde{T}$ requires careful tuning to balance theoretical accuracy with numerical stability. Future research will explore extending this framework to other gradient-flow variants beyond standard diffusion.

Conclusion

ScoreFusion provides a theoretically grounded and practically powerful tool for model ensembling. By recasting distribution fusion as a linear score-matching problem, it enables the creation of diverse, high-fidelity generative models in data-starved regimes, offering a significant improvement over heuristic merging techniques.

Summary updated

Abstract

We introduce ScoreFusion, a theoretically grounded method for fusing multiple pre-trained diffusion models that are assumed to generate from auxiliary populations. ScoreFusion is particularly useful for enhancing the generative modeling of a target population with limited observed data. Our starting point considers the family of KL barycenters of the auxiliary populations, which is proven to be an optimal parametric class in the KL sense, but difficult to learn. Nevertheless, by recasting the learning problem as score matching in denoising diffusion, we obtain a tractable way of computing the optimal KL barycenter weights. We prove a dimension-free sample complexity bound in total variation distance, provided that the auxiliary models are well-fitted for their own task and the auxiliary tasks combined capture the target well. The sample efficiency of ScoreFusion is demonstrated by learning handwritten digits. We also provide a simple adaptation of a Stable Diffusion denoising pipeline that enables sampling from the KL barycenter of two auxiliary checkpoints; on a portrait generation task, our method produces faces that enhance population heterogeneity relative to the auxiliary distributions.
BibTeX
@inproceedings{liu2025scorefusion,
  author = {Hao Liu and Tony Junze Ye and Jose Blanchet and Nian Si},
  title = {{ScoreFusion: Fusing Score-based Generative Models via Kullback–Leibler Barycenters}},
  booktitle = {International Conference on Artificial Intelligence and Statistics},
  year = {2025},
  url = {https://openreview.net/forum?id=KRQdmPQsvM},
}
Figure