Towards Privacy-Preserving Split Learning for ControlNet

Dixi YaoWACV 2025

TL;DR

This paper introduces a novel split learning framework for fine-tuning ControlNet models that significantly enhances efficiency and privacy. By eliminating server-to-client gradient transmission and employing privacy-preserving techniques for timestep sampling, activation functions, and prompt handling, the authors achieve robust privacy guarantees without sacrificing image generation quality.

What Problem Does This Paper Address?

Large generative models like diffusion models have become incredibly powerful, but fine-tuning them for specific use cases often requires users to provide their own data. This raises significant privacy concerns, especially when the data might contain personal information. ControlNet, a recent advancement, allows for more fine-grained control over image generation using conditions beyond text prompts. However, fine-tuning ControlNet with user-specific data, which might be distributed across multiple devices, presents a challenge: how to achieve this fine-tuning while ensuring the privacy of user data (images, conditions, and text prompts) and maintaining high-quality image generation. Existing split learning approaches, while offering privacy, can be inefficient and vulnerable to attacks.

Methods

The authors propose a multi-pronged approach to address the privacy and efficiency challenges of fine-tuning ControlNet using split learning:

  • Novel Split Learning Structure: They redesign the split learning architecture to eliminate the need for the server to send gradients back to the clients. This significantly reduces sequential dependencies and communication overhead, leading to faster training.
  • Privacy-Preserving Timestep Sampling: Inspired by the forward diffusion process, they propose a new timestep sampling policy. This policy leverages the relationship between the privacy budget ($\epsilon$) and the timestep, allowing for adjustable privacy guarantees. They mathematically derive how to set privacy budgets by selecting appropriate timesteps and scheduling parameters.
  • Noise-Confounding Activation Function: To prevent attackers from reconstructing private data from intermediate features, they introduce a novel activation function. This function, applied before features are sent to the server, adds a layer of obfuscation without degrading image generation quality.
  • Prompt-Hiding Fine-Tuning: Recognizing that text prompts can also contain private information, they propose a mechanism to fine-tune ControlNet without sending text prompts to the server. Only the client-side encoder block receives text prompts, and other parts of the network are adapted to work without them, preserving prompt privacy.

Datasets

  • Training: MS-COCO was used as the training dataset for fine-tuning diffusion models.
  • Evaluation: The evaluation involved generating images under various conditions, including Canny lines, scribbles, and segmentation maps, using datasets like CelebA and ImageNet.

Key Insights and Ideas

  • Efficiency Bottleneck in Split Learning: The paper highlights that the sequential nature of gradient transmission in traditional split learning is a major bottleneck for efficiency. Their proposed structure directly addresses this by removing the need for server-to-client gradient updates.
  • Timestep as a Privacy Lever: The insight that the privacy budget in diffusion models can be directly linked to the timestep offers a flexible and tunable privacy mechanism. This allows users to balance privacy and performance based on their needs.
  • Tailored Privacy for Generative Models: Unlike general-purpose privacy solutions, their methods are specifically designed for the unique characteristics of diffusion models and ControlNet, leading to better performance and privacy preservation.
  • Prompt Privacy is Crucial: The paper emphasizes that protecting text prompts is as important as protecting image data, and their prompt-hiding mechanism is a novel contribution in this regard.

Key Results

  • Efficiency Improvement: The proposed structure significantly reduces transmission overhead. For example, transmission overhead was reduced from 559.17 GB to 186.56 GB in their experiments.
  • Privacy Preservation: The methods demonstrate strong privacy protection, as evidenced by high PSNR and SSIM values for reconstructed images when privacy-preserving mechanisms are applied, indicating less similarity to original private data. For instance, they show a reduction in PSNR from 11.80 to 1.68 for Canny conditions.
  • Image Generation Quality: The proposed techniques maintain high image generation quality, comparable to production-level ControlNet, with minimal degradation in FID and CLIP scores across various conditions.
  • Resource Efficiency: Clients require less than 3 GB of GPU memory, making it feasible for a wider range of devices.

Limitations and Future Work

  • Attack Vulnerability Analysis: While the paper focuses on specific threats (inversion attacks and text prompt leakage), it acknowledges that other attacks might exist. The effectiveness of their defenses against a broader range of adversarial attacks could be further explored.
  • Hyperparameter Tuning: The paper mentions that the choice of ts (start timestep) is crucial for balancing privacy and performance, and that hyperparameter tuning is detailed in an appendix. Further research could explore automated or more robust hyperparameter selection strategies.
  • Generalizability: While the methods are tailored for ControlNet, exploring their applicability to other conditional generative models could be a future direction.
  • Real-world Deployment Complexity: While the paper mentions using PLATO for decentralized training, the practical challenges of deploying and managing such systems in real-world scenarios could be a subject for future work.

Summary updated

BibTeX
@inproceedings{yao2025towards,
  author = {Dixi Yao},
  title = {{Towards Privacy-Preserving Split Learning for ControlNet}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2025},
}
Figure