Conditional generative models face a fundamental tug-of-war between fidelity (how closely the output adheres to the conditioning prompt) and diversity or realism (how varied and natural the outputs are). As shown by Pietro Astolfi et al. in Consistency-diversity-realism Pareto fronts of conditional image generative models, navigating this multi-objective space typically requires shifting a model's position along a Pareto front.
The standard tool for traversing this front is Classifier-Free Guidance (CFG). By adjusting a post-hoc guidance scale, users can choose whether to prioritize strict prompt adherence or sample variety. This mechanism is highly versatile, controlling trade-offs in tasks ranging from black-box face inversion (Manuel Kansy et al. in Controllable Inversion of Black-Box Face Recognition Models via Diffusion), to detailed image captioning (Simon Kornblith et al. in Classifier-free guidance makes image captioning models more descriptive), to event boundary detection (Jaejun Hwang et al. in Generic Event Boundary Detection via Denoising Diffusion).
However, traditional CFG comes with a steep computational penalty. At every generation step, the model must execute two separate forward passes—one conditional and one unconditional—and linearly extrapolate between them:
$$\epsilon_{\text{guided}} = \epsilon_{\text{uncond}} + w \cdot (\epsilon_{\text{cond}} - \epsilon_{\text{uncond}})$$
This doubles the inference cost, creating a direct conflict between user controllability and computational efficiency. To resolve this, a new paradigm has emerged: internalizing the guidance scale as a continuous coordinate embedding directly inside the model.
Internalizing the Guidance Scale
To understand why this approach works, consider how diffusion models handle time. A single neural network can denoise an image at any stage of the generation process because the continuous timestep $t$ is transformed via Fourier or sinusoidal embeddings and injected into the network's layers.
By treating the guidance scale $w$ exactly like the timestep $t$, the network can learn a continuous manifold of guidance strengths natively. Instead of performing two forward passes and computing an algebraic extrapolation externally, the model accepts $w$ as a direct input coordinate.
Traditional CFG (Two Passes):
[Condition] ---> [Model] ---> \
---> Extrapolation ---> Guided Output
[Unconditioned] -> [Model] ---> /
Continuous Guidance Embedding (Single Pass):
[Condition] ---> \
[Guidance w] ---> ---> [Model (with w-embedding)] ---> Guided Output
Chenlin Meng et al. pioneered this concept in On Distillation of Guided Diffusion Models. By representing the guidance strength as a continuous coordinate using Fourier embeddings, they trained a single student model to emulate the combined output of a teacher model's dual passes. Because the guidance scale remains an adjustable input coordinate, the distilled model retains the ability to dynamically trade off diversity and prompt alignment at runtime, without needing separate models for different guidance strengths.
Single-Pass Inference and Distillation
Integrating the guidance scale as a continuous parameter allows the computational benefits of model distillation to be fully realized. When compressing multi-step diffusion processes into rapid, few-step or single-step generators, embedding the guidance scale ensures that controllability is not lost during the squeeze.
For example, Yatong Bai et al. developed ConsistencyTTA: Accelerating Diffusion-Based Text-to-Audio Generation with Consistency Distillation, which applies guidance-aware consistency distillation to text-to-audio models. By treating the guidance scale as a variable input during training, their single-step model retains the flexible, user-controlled creative fidelity that normally requires dozens of iterative passes.
A similar formulation solves the "perception-distortion" trade-off in image super-resolution. Runyi Li et al. in CTSR: Controllable Fidelity-Realness Trade-off Distillation for Real-World Image Super Resolution and Yuanzhi Zhu et al. in One-Step Flow for Image Super-Resolution with Tunable Fidelity-Realism Trade-offs demonstrated that a continuous hyperparameter can be integrated into a single-step network's forward pass. Sliding this parameter interpolates the inference path vector directly, allowing real-time adjustment from pixel-perfect reconstruction accuracy to rich, generative detail.
Guidance-Free Training from Scratch
This parameterization strategy is not limited to distilling pre-existing teachers; it can be baked directly into the primary training objective from scratch. This eliminates the redundant forward passes of post-hoc methods entirely.
In Diffusion Models without Classifier-free Guidance, Zhicong Tang et al. introduced Model-Guidance (MG). By treating the guidance scale as a dynamic input parameter during training, the model natively learns to generalize across different levels of prompt alignment. Similarly, Huayu Chen et al. proposed Guidance-Free Training (GFT) in Visual Generation Without Guidance. They feed a scalar parameter representing the guidance strength as an additional input to a single set of weights, mimicking the behavior of temperature scaling in language models.
This approach yields a double benefit: it cuts inference sampling time in half by requiring only a single forward pass, and it significantly accelerates training convergence. The same mechanism proves highly effective for efficient transfer learning; Yara Bahram et al. introduced DogFit: Domain-guided Fine-tuning for Efficient Transfer Learning of Diffusion Models, which injects a domain-aware guidance signal into the training loss as a conditional scalar input, mapping out a continuous manifold of guidance strengths that can be navigated at runtime in a single pass.
Alternative Trajectory Control Mechanisms
While continuous parameter injection represents the dominant trend for internalizing guidance, other methods manipulate generation trajectories through geometric or architectural constraints:
- Temporal Rigidity Control: Rather than keeping guidance constant, Shaogang Gong et al. show in Few-Shot Image Generation by Conditional Relaxing Diffusion Inversion that varying the influence of guidance embeddings across diffusion timesteps balances reconstruction fidelity and source-domain priors. High rigidity early on ensures structural alignment, while relaxation in later steps allows the model's priors to fill in diverse details.
- Orthogonal Trajectory Deviation: To increase diversity without losing prompt alignment, Jingxuan Wu et al. developed OSCAR: Orthogonal Stochastic Control for Alignment-Respecting Diversity in Flow Matching. This training-free method introduces a geometrically constrained control signal that pushes sampling trajectories to diverge orthogonally to the prompt alignment vector, spreading out the outputs in feature space without degrading quality.
- Dual-Embedding Regulation: In DreamArtist: Controllable One-Shot Text-to-Image Generation via Positive-Negative Adapter, ZiYi Dong et al. split the conditioning into positive and negative learned embeddings. The negative component acts as a critical regulator, bounding the generation trajectory to prevent overfitting to a single reference image while preserving output variety.
- Gradient-Based Steering: Beyond text prompts, external models can steer trajectories dynamically. Anh Nguyen et al. demonstrate this in Controllable Group Choreography using Contrastive Diffusion, using the gradients of a pre-trained contrastive encoder during the denoising phase to balance group synchronization against individual dancer diversity.
Ultimately, whether achieved through continuous coordinate embeddings, temporal scheduling, or geometric constraints, moving guidance from an external post-hoc patch to an intrinsic architectural parameter allows generative models to remain highly controllable while running twice as fast.
Key Foundational Work
On Distillation of Guided Diffusion Models (2022) — contributed 3 concept elements to this direction.
Controllable Inversion of Black-Box Face-Recognition Models via Diffusion (2023) — contributed 1 concept elements to this direction.
Classifier-free guidance makes image captioning models more descriptive (2023) — contributed 1 concept elements to this direction.