CCM: Real-Time Controllable Visual Content Creation Using Text-to-Image Consistency Models

Jie Xiao, Kai Zhu, Han Zhang, Zhiheng Liu, Yujun Shen, Zhantao Yang, Ruili Feng, Yu Liu, Xueyang Fu, Zheng-Jun Zha

Introduction

Diffusion Models (DMs) like Stable Diffusion currently dominate AI image generation. Tools like ControlNet have made them highly controllable, allowing users to guide generation using spatial conditions such as poses, edges, or depth maps. However, DMs are inherently slow because they require dozens of iterative denoising steps.

Recently, Consistency Models (CMs) have emerged as a faster alternative, capable of generating high-quality images in just 1 to 4 steps. But there is a catch: the rich ecosystem of conditional controls (like ControlNet) was built for DMs, not CMs. If a user wants to generate an image matching a specific sketch in real-time using a CM, the existing DM-based ControlNets cannot simply be plugged in without issues. This paper explores how to bridge the gap between the generative speed of CMs and the fine-grained spatial controllability of ControlNet, enabling real-time, highly controllable image generation that matches diffusion model quality at a fraction of the computational cost.

Methods: Integrating ControlNet with Consistency Models

The authors investigate three distinct strategies for adding ControlNet to a pre-trained text-to-image Consistency Model:

Overview of the three strategies for integrating ControlNet with Consistency Models.
  1. Direct Transfer (The Naive Approach): Because CMs and DMs share the same underlying mathematical trajectory (the Probability Flow ODE), the authors first tried simply plugging a pre-trained DM ControlNet directly into a CM. While it successfully passes high-level semantic layout information without retraining, it fails at low-level details, resulting in unrealistic or artifact-heavy images.
  2. Consistency Training for ControlNet (The Optimal Approach): Instead of relying on DM assets, the authors treat the frozen CM and a newly initialized ControlNet as a single, new conditional CM. They train the ControlNet from scratch using consistency training—enforcing that the model produces consistent outputs regardless of which point on the noise trajectory it starts from.
  3. Unified Adapter (The Pragmatic Approach): Training ControlNets from scratch is computationally expensive. To leverage the massive existing ecosystem of DM ControlNets, the authors propose training a lightweight, unified adapter. Using consistency training, this adapter learns to "translate" the features from a frozen DM ControlNet so they align properly with the CM, mitigating the architectural gap between the two.

Training and Datasets

To train the ControlNets and the foundational CM, the authors utilized a combination of public datasets, including ImageNet21K, WebVision, and a filtered version of the LAION dataset (removing duplicates, low-res, and harmful content). For the conditional inputs, they generated synthetic labels using standard pre-trained extractors: MiDaS for depth, Canny/HED for edges, human pose skeletons, random masking for inpainting, and 16x downscaling for super-resolution.

Key Insights and Results

The quantitative and qualitative results strongly validate the CM-native approaches, proving that consistency training is highly versatile. The consistency loss is not just for distilling base models; it is highly effective for training auxiliary control networks from scratch or adapting existing ones.

Performance and efficiency comparison of one-step generation across different ControlNet integration strategies.
  • Massive Speedups: The baseline DM ControlNet requires 50 steps (Number of Function Evaluations, or NFEs) and takes ~23.6 seconds to generate a 1024x1024 image. The CM ControlNet requires only 1 to 4 steps, taking just 0.2 to 0.9 seconds—a roughly 26x to 100x speedup.
  • Competitive Quality: Using 4 steps, the natively trained CM ControlNet achieves an average FID of 7.61 (measuring image quality) and a Fidelity score of 0.130 (measuring adherence to the control signal). This is highly competitive with the much slower DM baseline (FID 7.31, Fidelity 0.122).
  • Adapter Effectiveness: The unified adapter is a massive win for practicality. By training one shared adapter across multiple conditions, users can plug-and-play existing DM ControlNets into CMs. For 1-step generation, the naive "Direct Transfer" method yields a poor FID of 23.19. Adding the Unified Adapter drastically improves this to 16.96, proving the adapter successfully bridges the gap between DM controls and CM generation without the massive compute cost of training new CM-specific ControlNets.

Limitations and Future Work

While the adapter method is highly efficient, it still lags slightly behind the natively trained CM ControlNet in terms of raw image quality and detail adherence. Future technical work will likely focus on closing this gap, allowing for perfect zero-shot transfer of DM assets to real-time CM architectures.

Furthermore, the authors acknowledge the societal implications of this technology. The ability to generate highly controllable, hyper-realistic images instantaneously significantly lowers the barrier for creating deepfakes and propagating visual misinformation, necessitating robust safeguards as these real-time models become widely available.

Summary updated

Abstract

Consistency Models (CMs) have showed a promise in creating high-quality images with few steps. However, the way to add new conditional controls to the pre-trained CMs has not been explored. In this paper, we explore the pivotal subject of leveraging the generative capacity and efficiency of consistency models to facilitate controllable visual content creation via ControlNet. First, it is observed that ControlNet trained for diffusion models (DMs) can be directly applied to CMs for high-level semantic controls but sacrifice image low-level details and realism. To tackle with this issue, we develop a CMs-tailored training strategy for ControlNet using the consistency training. It is substantiated that ControlNet can be successfully established through the consistency training technique. Besides, a unified adapter can be trained utilizing the consistency training, which enhances the adaptation of DM's ControlNet. We quantitatively and qualitatively evaluate all strategies across various conditional controls, including sketch, hed, canny, depth, human pose, low-resolution image and masked image, with the pre-trained text-to-image latent consistency models.
BibTeX
@inproceedings{xiao2024ccm,
  author = {Jie Xiao and Kai Zhu and Han Zhang and Zhiheng Liu and Yujun Shen and Zhantao Yang and Ruili Feng and Yu Liu and Xueyang Fu and Zheng-Jun Zha},
  title = {{CCM: Real-Time Controllable Visual Content Creation Using Text-to-Image Consistency Models}},
  booktitle = {International Conference on Machine Learning},
  year = {2024},
  url = {https://openreview.net/forum?id=B4rViOCoNf},
}
Figure