Continuous Guidance Embeddings for Controllable Generation

21 concepts15 papers

Conditional generative models face a fundamental tug-of-war between fidelity (how closely the output adheres to the conditioning prompt) and diversity or realism (how varied and natural the outputs are). As shown by Pietro Astolfi et al. in Consistency-diversity-realism Pareto fronts of conditional image generative models, navigating this multi-objective space typically requires shifting a model's position along a Pareto front.

The standard tool for traversing this front is Classifier-Free Guidance (CFG). By adjusting a post-hoc guidance scale, users can choose whether to prioritize strict prompt adherence or sample variety. This mechanism is highly versatile, controlling trade-offs in tasks ranging from black-box face inversion (Manuel Kansy et al. in Controllable Inversion of Black-Box Face Recognition Models via Diffusion), to detailed image captioning (Simon Kornblith et al. in Classifier-free guidance makes image captioning models more descriptive), to event boundary detection (Jaejun Hwang et al. in Generic Event Boundary Detection via Denoising Diffusion).

However, traditional CFG comes with a steep computational penalty. At every generation step, the model must execute two separate forward passes—one conditional and one unconditional—and linearly extrapolate between them:

$$\epsilon_{\text{guided}} = \epsilon_{\text{uncond}} + w \cdot (\epsilon_{\text{cond}} - \epsilon_{\text{uncond}})$$

This doubles the inference cost, creating a direct conflict between user controllability and computational efficiency. To resolve this, a new paradigm has emerged: internalizing the guidance scale as a continuous coordinate embedding directly inside the model.

Internalizing the Guidance Scale

To understand why this approach works, consider how diffusion models handle time. A single neural network can denoise an image at any stage of the generation process because the continuous timestep $t$ is transformed via Fourier or sinusoidal embeddings and injected into the network's layers.

By treating the guidance scale $w$ exactly like the timestep $t$, the network can learn a continuous manifold of guidance strengths natively. Instead of performing two forward passes and computing an algebraic extrapolation externally, the model accepts $w$ as a direct input coordinate.

Traditional CFG (Two Passes):
[Condition]   ---> [Model] ---> \
                                 ---> Extrapolation ---> Guided Output
[Unconditioned] -> [Model] ---> /

Continuous Guidance Embedding (Single Pass):
[Condition]   ---> \
[Guidance w]  --->  ---> [Model (with w-embedding)] ---> Guided Output

Chenlin Meng et al. pioneered this concept in On Distillation of Guided Diffusion Models. By representing the guidance strength as a continuous coordinate using Fourier embeddings, they trained a single student model to emulate the combined output of a teacher model's dual passes. Because the guidance scale remains an adjustable input coordinate, the distilled model retains the ability to dynamically trade off diversity and prompt alignment at runtime, without needing separate models for different guidance strengths.

Single-Pass Inference and Distillation

Integrating the guidance scale as a continuous parameter allows the computational benefits of model distillation to be fully realized. When compressing multi-step diffusion processes into rapid, few-step or single-step generators, embedding the guidance scale ensures that controllability is not lost during the squeeze.

For example, Yatong Bai et al. developed ConsistencyTTA: Accelerating Diffusion-Based Text-to-Audio Generation with Consistency Distillation, which applies guidance-aware consistency distillation to text-to-audio models. By treating the guidance scale as a variable input during training, their single-step model retains the flexible, user-controlled creative fidelity that normally requires dozens of iterative passes.

A similar formulation solves the "perception-distortion" trade-off in image super-resolution. Runyi Li et al. in CTSR: Controllable Fidelity-Realness Trade-off Distillation for Real-World Image Super Resolution and Yuanzhi Zhu et al. in One-Step Flow for Image Super-Resolution with Tunable Fidelity-Realism Trade-offs demonstrated that a continuous hyperparameter can be integrated into a single-step network's forward pass. Sliding this parameter interpolates the inference path vector directly, allowing real-time adjustment from pixel-perfect reconstruction accuracy to rich, generative detail.

Guidance-Free Training from Scratch

This parameterization strategy is not limited to distilling pre-existing teachers; it can be baked directly into the primary training objective from scratch. This eliminates the redundant forward passes of post-hoc methods entirely.

In Diffusion Models without Classifier-free Guidance, Zhicong Tang et al. introduced Model-Guidance (MG). By treating the guidance scale as a dynamic input parameter during training, the model natively learns to generalize across different levels of prompt alignment. Similarly, Huayu Chen et al. proposed Guidance-Free Training (GFT) in Visual Generation Without Guidance. They feed a scalar parameter representing the guidance strength as an additional input to a single set of weights, mimicking the behavior of temperature scaling in language models.

This approach yields a double benefit: it cuts inference sampling time in half by requiring only a single forward pass, and it significantly accelerates training convergence. The same mechanism proves highly effective for efficient transfer learning; Yara Bahram et al. introduced DogFit: Domain-guided Fine-tuning for Efficient Transfer Learning of Diffusion Models, which injects a domain-aware guidance signal into the training loss as a conditional scalar input, mapping out a continuous manifold of guidance strengths that can be navigated at runtime in a single pass.

Alternative Trajectory Control Mechanisms

While continuous parameter injection represents the dominant trend for internalizing guidance, other methods manipulate generation trajectories through geometric or architectural constraints:

Ultimately, whether achieved through continuous coordinate embeddings, temporal scheduling, or geometric constraints, moving guidance from an external post-hoc patch to an intrinsic architectural parameter allows generative models to remain highly controllable while running twice as fast.

Key Foundational Work

On Distillation of Guided Diffusion Models (2022) — contributed 3 concept elements to this direction.
Controllable Inversion of Black-Box Face-Recognition Models via Diffusion (2023) — contributed 1 concept elements to this direction.
Classifier-free guidance makes image captioning models more descriptive (2023) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_parameterized_guidance_conditioning_80f1d4cd,
  title = {parameterized guidance conditioning},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/parameterized-guidance-conditioning/rdr_80f1d4cd7b2149ef8e3a6c5ff8fb6f13},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-21},
}

This exact version (version 1, frozen text):

@misc{lacuna_parameterized_guidance_conditioning_80f1d4cd_v1,
  title = {parameterized guidance conditioning},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/parameterized-guidance-conditioning/rdr_80f1d4cd7b2149ef8e3a6c5ff8fb6f13/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-21},
}

See something wrong? Report a mistake