Diffusion models generate high-quality data by iteratively nudging noise toward a learned data manifold. However, when we try to "steer" this process—whether to follow a text prompt, solve an inverse problem, or satisfy physical constraints—we often run into a fundamental geometric conflict. Standard guidance techniques frequently push the sampling trajectory off the manifold into "impossible" regions, resulting in the oversaturated, "deep-fried" artifacts and hallucinations common at high guidance scales.
A growing body of research suggests that the key to stable, high-fidelity steering is manifold preservation: ensuring that external guidance updates remain within the local tangent space of the data distribution.
The Geometry of Guidance Failure
To understand why guidance fails, we must look at the score function. In a standard diffusion model, the score function $\nabla_x \log p(x)$ provides a vector field that points toward the high-density regions of the data manifold. Guidance adds an external signal, such as a classifier gradient $\nabla_x \log p(y|x)$ or a reward gradient.
As Hyungjin Chung and Jeongsol Kim observe in CFG++: Manifold-constrained Classifier Free Guidance for Diffusion Models, standard Classifier-Free Guidance (CFG) relies on linear extrapolation. When the guidance scale is high, this extrapolation pushes the latent state into low-density regions where the model was never trained. In these "off-manifold" regions, the model's internal score function becomes inaccurate, leading to the collapse of image structure and extreme pixel values.
Cheng Jin and Chutao Liu identify a similar phenomenon in Angle Domain Guidance, noting that high-magnitude updates cause "anomalous diffusion" where the update direction points toward regions of lower probability density.
Projecting onto the Tangent Space
The solution proposed across several papers is to decompose the guidance gradient into two components: one tangential to the manifold and one normal (perpendicular) to it.
In Manifold Preserving Guided Diffusion, Yutong He and Naoki Murata argue that updates should be restricted to the tangent space. By projecting the guidance gradient onto this space, we ensure the update moves the sample along the surface of the manifold—changing its attributes (like color or shape) without pushing it into the "void" of unrealistic noise.
This geometric intuition is shared by Dohoon Ryu in Improving Diffusion Models for Inverse Problems using Manifold Constraints. Ryu demonstrates that while the score function provides guidance normal to the manifold (keeping the image "real"), effective problem-solving requires additional guidance along the tangent space to satisfy external measurements without sacrificing that realism.
Mechanisms for Manifold Alignment
Researchers have developed several "plug-and-play" methods to enforce these constraints:
- Tangential Damping: Mingi Kwon and colleagues introduced TCFG: Tangential Damping Classifier-free Guidance, which uses Singular Value Decomposition (SVD) to identify and dampen components of the guidance signal that are misaligned with the target manifold.
- Jacobian Projection: In How to Guide Your Flow: Few-Step Alignment via Flow Map Reward Guidance, Jerry Yuyang Huang and Justin Lin use the Jacobian of the flow map as a projector. This filters out reward gradients that would otherwise push the sample off-manifold, enabling steering in as few as three steps.
- Local Neighborhoods: For offline reinforcement learning, Kyowoon Lee and Jaesik Choi developed LoMAP, which retrieves neighbors from a training dataset to approximate the local manifold, projecting intermediate diffusion steps back onto a valid subspace to prevent "physically impossible" trajectories.
Beyond Image Generation
The importance of manifold preservation extends far beyond aesthetic image editing. It is a critical requirement for safety and physical validity in scientific and robotic applications.
In robotics, Hao Ren and Zetong Bi developed FPG-OPS (Fisher-Preserving Guidance: Training-Free Manifold Constraints for Safe Diffusion Control). They found that unconstrained task guidance (like collision avoidance) often causes "manifold explosion," where the robot's policy drifts into unreliable action spaces. Projecting these updates onto a sensitivity manifold maintains numerical stability and ensures the robot's actions remain within the distribution of safe, learned behaviors.
Similarly, in quantum chemistry, John C. Snyder and Klaus Robert Muller showed in Understanding Machine-learned Density Functionals that gradient descent for electron density fails unless gradients are projected onto the tangent space of physically valid states.
Summary of Intuition
The core insight of this research direction is that the learned data distribution is a thin "shell" in a high-dimensional space. Standard guidance acts like a force applied to a particle on this shell. If the force is too strong or poorly aimed, it knocks the particle off the shell into a vacuum where the model's "physics" no longer apply. By mathematically constraining these forces to act only along the surface of the shell, we can steer the generative process toward specific goals while maintaining the structural integrity and realism of the data.