CollaFuse: Navigating Limited Resources and Privacy in Collaborative Generative AI

Domenique Zipperling, Simeon Allmendinger, Lukas Struppek, Niklas Kühl

Introduction

Generative AI, particularly Denoising Diffusion Probabilistic Models (DDPMs), has revolutionized high-fidelity image synthesis. However, these models impose significant "entry fees": high computational costs and stringent data privacy requirements. Training and inference demand high-end GPUs, which are often unavailable on edge devices like medical scanners. Furthermore, in fields like healthcare, sharing raw sensitive data with a central server is frequently prohibited by legal and ethical frameworks.

While Federated Learning (FL) offers a decentralized training approach, it still places a heavy computational burden on the local "client" device. CollaFuse addresses this bottleneck by partitioning the workload between local clients and a shared server, optimizing for both resource efficiency and data privacy.

The CollaFuse Framework

CollaFuse is inspired by Split Learning (SL), where a model's architecture is divided. In this framework, the workload is partitioned based on a cut-ratio ($c$), which determines the balance of tasks between the client and the server.

The CollaFuse framework architecture showing (a) the local denoising cycle and (b) the multi-client collaborative setup.

The process follows a six-step collaborative cycle:

  1. Local Diffusion: The client adds noise to its private data. This step is computationally inexpensive.
  2. Transmission: The client sends the noisy, unrecognizable image to a shared server.
  3. Server-Side Denoising: The server performs the computationally intensive initial denoising steps using a shared "backbone" model.
  4. Return: The server sends the partially denoised image back to the client.
  5. Local Completion: The client performs the final denoising steps locally to generate the final high-quality image.

By adjusting the cut-ratio ($c$), users can control the trade-off: a lower $c$ offloads more work to the server, while a higher $c$ keeps more processing (and privacy) local.

Experimental Setup

The researchers evaluated CollaFuse using a healthcare-focused scenario involving MRI brain scans from the BraTS initiative. The setup included three independent clients, each managing 4,920 MRI scans. A hold-out test set of 5,000 images from different patients was used to evaluate the model's generalization capabilities.

Key Results and Insights

The study revealed several critical findings regarding the efficacy of collaborative diffusion:

The "U-Shaped" Performance Curve

A significant discovery was that collaboration does more than just save resources—it improves image quality. The researchers observed a U-shaped pattern in image fidelity, measured by Kernel Inception Distance (KID) scores.

KID scores across different cut-ratios, illustrating the performance gains from collaborative training.

Collaborative training often outperformed purely local training ($c=1.0$). This suggests that the shared server backbone learns more robust features by aggregating noisy inputs from multiple clients. However, there is a "tipping point": if the cut-ratio is too low (e.g., $c=0$), performance may degrade, indicating an optimal "sweet spot" for model splitting.

Privacy and Efficiency

  • Privacy through Noise: Even when the server handles a large portion of the denoising, the images it receives are heavily noised. Analysis using Mean Squared Error (MSE) showed that identifiable features of the raw data remain concealed from the server during the majority of the process.
  • Resource Savings: Offloading denoising steps to the server directly reduces local GPU energy consumption, making advanced generative AI feasible for resource-constrained edge devices.

Limitations and Future Directions

As a "Short Paper" contribution, CollaFuse has several avenues for future development:

  • Data Complexity: Future work will expand from grayscale MRI scans to color imagery and more diverse datasets.
  • Text-to-Image Integration: The authors plan to incorporate text-driven generation (similar to Stable Diffusion) into the framework.
  • Formal Privacy Guarantees: While noise provides implicit privacy, implementing formal safeguards like Differential Privacy is a priority to protect against sophisticated reconstruction attacks.
  • Communication Overhead: Further analysis is needed to ensure that the bandwidth costs of data transmission do not outweigh the GPU energy savings.

Summary updated

BibTeX
@inproceedings{zipperling2024collafuse,
  author = {Domenique Zipperling and Simeon Allmendinger and Lukas Struppek and Niklas Kühl},
  title = {{CollaFuse: Navigating Limited Resources and Privacy in Collaborative Generative AI}},
  booktitle = {ECIS},
  year = {2024},
}
Figure