Introduction
Diffusion models, such as Stable Diffusion, have transformed generative AI but remain computationally intensive, requiring significant GPU resources for both training and inference. While Federated Learning (FL) offers a path toward collaborative training without raw data sharing, it typically requires clients to manage the entire model locally—a prohibitive requirement for resource-constrained edge devices.
CollaFuse introduces a "split learning" framework specifically designed for diffusion models. By partitioning the iterative denoising process between a powerful central server and multiple clients, CollaFuse democratizes access to high-quality generative models while preserving data privacy and reducing local hardware demands.
Methodology: Splitting the Denoising Process
Unlike traditional split learning that divides neural network layers, CollaFuse splits the denoising trajectory. In a standard diffusion process, an image is generated from pure noise ($x_T$) through $T$ iterative steps. CollaFuse introduces a cut point ($t_\zeta$) to divide this workload:
- Server-Side Denoising: A shared server model performs the initial, computationally heavy denoising steps from $T$ down to $t_\zeta$. At these early stages, the image is predominantly noise, which naturally masks sensitive local data features.
- Client-Side Refinement: The partially denoised intermediate representation is transmitted to the client. The client then uses a local, personalized model to complete the final denoising steps (from $t_\zeta$ down to 0), adding fine-grained details and specific stylistic characteristics.
This approach ensures that server and clients maintain independent weights ($\theta^s$ and $\theta^c$), eliminating the need for weight synchronization and reducing communication overhead. An adaptive noise scheduler is employed to ensure a seamless transition at the cut point.
Experimental Evaluation
The framework was evaluated across three diverse benchmarks: CelebA (facial attributes), CIFAR-10 (general objects), and Animals-with-Attributes2 (Awa2). These tests simulated environments with multiple clients possessing distinct local data distributions.
Key Performance Metrics
The researchers measured performance using Fréchet Inception Distance (FID) and Fréchet CLIP Distance (FCD) to assess image quality and semantic alignment.
As shown in the results, CollaFuse consistently outperforms Independent Client Models (ICMs)—models trained solely on local data. Remarkably, in several configurations, CollaFuse even surpassed the Global Model (GM), which is trained on the pooled data of all clients. This suggests that the split architecture effectively balances global knowledge with local personalization, avoiding the "one-size-fits-all" degradation often seen in centralized training.
Insights and Trade-offs
- Privacy through Stochasticity: The high noise level at the cut point serves as a privacy buffer. By the time the server hands off the image, the structural "blob" contains little identifiable information.
- The "Goldilocks" Cut Point: The choice of $t_\zeta$ is critical. If $t_\zeta$ is too high, the client bears too much computational load; if it is too low, the client loses the ability to personalize the output. Empirical results suggest that $t_\zeta \le 200$ (out of 1000 steps) provides an optimal balance of efficiency and quality.
- Resource Efficiency: By offloading up to 80% of the denoising steps to the server, clients can participate in state-of-the-art image synthesis with a fraction of the memory and power required for full-model training.
Conclusion and Future Directions
CollaFuse demonstrates that collaborative diffusion is not only possible but can lead to superior model performance compared to isolated training. While the current framework assumes "honest-but-curious" participants, future work aims to integrate formal Differential Privacy (DP) guarantees and explore robustness against potential model poisoning or backdoor attacks. Scaling this approach to multi-billion parameter foundation models remains a key frontier for democratizing generative AI.
Summary updated