Introduction
Diffusion Models (DMs) like Stable Diffusion currently dominate AI image generation. Tools like ControlNet have made them highly controllable, allowing users to guide generation using spatial conditions such as poses, edges, or depth maps. However, DMs are inherently slow because they require dozens of iterative denoising steps.
Recently, Consistency Models (CMs) have emerged as a faster alternative, capable of generating high-quality images in just 1 to 4 steps. But there is a catch: the rich ecosystem of conditional controls (like ControlNet) was built for DMs, not CMs. If a user wants to generate an image matching a specific sketch in real-time using a CM, the existing DM-based ControlNets cannot simply be plugged in without issues. This paper explores how to bridge the gap between the generative speed of CMs and the fine-grained spatial controllability of ControlNet, enabling real-time, highly controllable image generation that matches diffusion model quality at a fraction of the computational cost.
Methods: Integrating ControlNet with Consistency Models
The authors investigate three distinct strategies for adding ControlNet to a pre-trained text-to-image Consistency Model:
- Direct Transfer (The Naive Approach): Because CMs and DMs share the same underlying mathematical trajectory (the Probability Flow ODE), the authors first tried simply plugging a pre-trained DM ControlNet directly into a CM. While it successfully passes high-level semantic layout information without retraining, it fails at low-level details, resulting in unrealistic or artifact-heavy images.
- Consistency Training for ControlNet (The Optimal Approach): Instead of relying on DM assets, the authors treat the frozen CM and a newly initialized ControlNet as a single, new conditional CM. They train the ControlNet from scratch using consistency training—enforcing that the model produces consistent outputs regardless of which point on the noise trajectory it starts from.
- Unified Adapter (The Pragmatic Approach): Training ControlNets from scratch is computationally expensive. To leverage the massive existing ecosystem of DM ControlNets, the authors propose training a lightweight, unified adapter. Using consistency training, this adapter learns to "translate" the features from a frozen DM ControlNet so they align properly with the CM, mitigating the architectural gap between the two.
Training and Datasets
To train the ControlNets and the foundational CM, the authors utilized a combination of public datasets, including ImageNet21K, WebVision, and a filtered version of the LAION dataset (removing duplicates, low-res, and harmful content). For the conditional inputs, they generated synthetic labels using standard pre-trained extractors: MiDaS for depth, Canny/HED for edges, human pose skeletons, random masking for inpainting, and 16x downscaling for super-resolution.
Key Insights and Results
The quantitative and qualitative results strongly validate the CM-native approaches, proving that consistency training is highly versatile. The consistency loss is not just for distilling base models; it is highly effective for training auxiliary control networks from scratch or adapting existing ones.
- Massive Speedups: The baseline DM ControlNet requires 50 steps (Number of Function Evaluations, or NFEs) and takes ~23.6 seconds to generate a 1024x1024 image. The CM ControlNet requires only 1 to 4 steps, taking just 0.2 to 0.9 seconds—a roughly 26x to 100x speedup.
- Competitive Quality: Using 4 steps, the natively trained CM ControlNet achieves an average FID of 7.61 (measuring image quality) and a Fidelity score of 0.130 (measuring adherence to the control signal). This is highly competitive with the much slower DM baseline (FID 7.31, Fidelity 0.122).
- Adapter Effectiveness: The unified adapter is a massive win for practicality. By training one shared adapter across multiple conditions, users can plug-and-play existing DM ControlNets into CMs. For 1-step generation, the naive "Direct Transfer" method yields a poor FID of 23.19. Adding the Unified Adapter drastically improves this to 16.96, proving the adapter successfully bridges the gap between DM controls and CM generation without the massive compute cost of training new CM-specific ControlNets.
Limitations and Future Work
While the adapter method is highly efficient, it still lags slightly behind the natively trained CM ControlNet in terms of raw image quality and detail adherence. Future technical work will likely focus on closing this gap, allowing for perfect zero-shot transfer of DM assets to real-time CM architectures.
Furthermore, the authors acknowledge the societal implications of this technology. The ability to generate highly controllable, hyper-realistic images instantaneously significantly lowers the barrier for creating deepfakes and propagating visual misinformation, necessitating robust safeguards as these real-time models become widely available.
Summary updated