Velocity Prediction in Generative Diffusion Models

16 concepts10 papers1 proposal

Traditional diffusion models operate by transforming clean data into pure noise, and then learning to reverse this process. During training, a neural network is typically tasked with predicting one of two endpoints: either the original clean data $x_0$ or the random noise $\epsilon$ that was added to it.

While intuitive, this binary choice introduces severe numerical and optimization challenges at the boundaries of the diffusion process. If the model predicts noise, it struggles at the beginning of generation when the input is almost pure noise, because the signal-to-noise ratio (SNR) approaches zero. If it predicts the clean data, the loss landscape becomes highly unstable near the data distribution.

To resolve this dilemma, a powerful alternative has emerged: velocity prediction (often called $v$-prediction or velocity parameterization). Instead of predicting the static endpoints of the transformation, the model is trained to predict the velocity vector—the instantaneous direction and speed of the trajectory moving from noise to data.


The Geometry of the Velocity Target

To understand why velocity prediction works, consider how a data point $x_0$ is mixed with noise $\epsilon$ over time $t$. The intermediate latent state $x_t$ is typically expressed as a linear interpolation:

$$x_t = \alpha_t x_0 + \sigma_t \epsilon$$

Where $\alpha_t$ and $\sigma_t$ dictate the noise schedule. The velocity $v_t$ is simply the time-derivative of this path:

$$v_t = \frac{dx_t}{dt} = \dot{\alpha}_t x_0 + \dot{\sigma}_t \epsilon$$

This formulation reveals the core intuition behind the approach: velocity is a smoothly changing, weighted combination of both the data and the noise.

As originally shown by Salimans et al. and Ho in Progressive Distillation for Fast Sampling of Diffusion Models, predicting this velocity vector provides a unified target that gracefully transitions between noise-prediction and data-prediction depending on the timestep. Rather than forcing the network to make a hard conceptual pivot across different stages of the generation process, velocity parameterization treats the entire trajectory as a continuous flow field.


Eliminating Boundary Singularities

The most compelling mathematical argument for velocity prediction lies in how it behaves at the extreme edges of the diffusion process.

The Zero-SNR Problem

Standard diffusion models often use noise schedules that never fully reach a signal-to-noise ratio of zero. As exposed by Liu et al. and Lin in Common Diffusion Noise Schedules and Sample Steps are Flawed, this design flaw prevents models from generating images with full dynamic range, making truly dark or bright scenes impossible to synthesize. However, if you enforce a "zero terminal SNR" schedule using standard noise prediction, the objective blows up mathematically because the model tries to predict noise from pure noise, creating a singularity. Switching to a velocity target resolves this completely: when the signal is entirely absent, the velocity vector remains bounded and stable, pointing cleanly toward the data distribution.

The Data-Distribution Singularity

A reciprocal issue occurs at the other end of the timeline ($t \to 0$), where the latent state is nearly clean data. Score-based methods often encounter mathematical singularities here because the score function (the gradient of the log-density) diverges as the distribution sharpens. As explored by Ma et al. et al. with Scalable Interpolant Transformers in SiT, velocity-based objectives bypass these singularities entirely. The velocity remains well-behaved throughout the entire transition, ensuring stable training from the first step to the last.


Balancing the Optimization Landscape

In standard diffusion training, the loss function can be highly unbalanced. At certain timesteps, predicting the target is trivial; at others, it is incredibly difficult. This imbalance forces the network to over-allocate its capacity to specific noise regimes while neglecting others.

By reparameterizing the network to target velocity, the learning objective is naturally balanced across all noise levels. In Improved Techniques for Maximum Likelihood Estimation for Diffusion ODEs, Zheng et al. et al. demonstrated that this variance-balancing property prevents the model from over-focusing on regions with extreme SNRs. Consequently, models trained with velocity parameterization converge 2 to 3 times faster than traditional methods while achieving significantly better density estimation accuracy.


Accelerating Inference and Distillation

Beyond training stability, modeling the velocity field yields smoother trajectories. When the underlying vector field is smooth, numerical ordinary differential equation (ODE) solvers can take much larger steps without drifting off course. Zhang et al. and Chen in Fast Sampling of Diffusion Models with Exponential Integrator note that a smoother objective over the sampling trajectory directly facilitates more accurate higher-order numerical approximations, enabling high-quality generation in fewer steps.

This smoothness is particularly vital for model distillation, where a pre-trained model is compressed into a fast, few-step sampler. When a model is forced to make massive updates in a single step, pixel-space or noise-space errors can compound catastrophically.

Standard Space (Noise/Pixel):
[Pure Noise] ----(Large Step Error)----> [Distorted Latent] ----> [Degraded Image]

Velocity Space:
[Pure Noise] ----(Stable Direction)----> [Smooth Trajectory] ----> [High-Fidelity Image]

Distilling in velocity space focuses the student network on learning the overall direction and speed of the flow field rather than direct pixel reconstruction. As demonstrated by Cai et al. et al. in Shortcutting Pre-trained Flow Matching Diffusion Models is Almost Free Lunch, velocity-space distillation provides superior numerical stability and faster convergence because the magnitude of errors allows for more effective gradient updates than the tiny, fractional errors found in pixel space.


Broadening Applications: From Particles to Motion

The conceptual shift from "predicting noise" to "modeling a velocity field" has allowed researchers to apply diffusion mechanics to domains far beyond image synthesis.

Key Foundational Work

Progressive Distillation for Fast Sampling of Diffusion Models (2022) — contributed 2 concept elements to this direction.
Improved Techniques for Maximum Likelihood Estimation for Diffusion ODEs (2023) — contributed 2 concept elements to this direction.
Fast Sampling of Diffusion Models with Exponential Integrator (2023) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_velocity_based_diffusion_objective_abc94a24,
  title = {velocity-based diffusion objective},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/velocity-based-diffusion-objective/rdr_abc94a24c4a64ec09ae54271b21a9358},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-21},
}

This exact version (version 1, frozen text):

@misc{lacuna_velocity_based_diffusion_objective_abc94a24_v1,
  title = {velocity-based diffusion objective},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/velocity-based-diffusion-objective/rdr_abc94a24c4a64ec09ae54271b21a9358/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-21},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake