Standard diffusion models, such as those widely used for image generation, operate by predicting the noise added to a sample at any given timestep. This noise-prediction ($\epsilon$-prediction) framework works remarkably well for pixels because high-frequency Gaussian noise is statistically distinct from natural image structures. However, as diffusion models have expanded into domains like human motion synthesis, time series forecasting, and audio enhancement, this paradigm has encountered fundamental limitations.
Instead of predicting the noise vector, a powerful alternative approach parameterizes the underlying neural network to directly predict the clean target signal ($x_0$-prediction) at each denoising step. Shifting the optimization objective from noise-space to signal-space fundamentally changes how a model handles physical constraints, preserves sequential continuity, and behaves during fast inference.
Enforcing Physical and Geometric Constraints
Why does predicting the final clean signal matter so much for structured data like human animation or physical fields? When a network outputs a noise vector $\epsilon$, it is mathematically isolated from the physical reality of the actual sample. It is incredibly difficult to compute whether a predicted noise vector will ultimately cause a 3D character's feet to slide through the floor or violate anatomical joint limits.
By shifting the network's objective to predict the clean motion sample directly at each step, the model can apply explicit geometric and physical loss functions during training. In their seminal work on the Human Motion Diffusion Model, Guy Tevet et al. demonstrated that direct $x_0$-prediction allows the model to optimize for joint positions, velocities, and foot-contact constraints throughout the entire denoising process. This direct feedback loop prevents common animation artifacts and ensures anatomical plausibility.
Noise Prediction (ε-parameterization):
Model Output (ε) ──> Hard to constrain physically ──> Loss computed on noise
Direct Signal Prediction (x0-parameterization):
Model Output (x0) ──> Kinematic/Geometric Losses ──> Highly realistic structure
(Foot contact, joint velocity)
This structural grounding is critical across various motion-generation tasks:
- Skeletal Integrity: Jiarui Sun 0001 et al. leveraged this property in CoMusion: Towards Consistent Stochastic Human Motion Prediction via Motion Diffusion to incorporate explicit joint and skeletal constraints into the optimization loop, maintaining physical consistency over time.
- Kinematic Refinement: In the context of zero-shot text-to-motion generation, Xu Shi et al. augmented direct signal prediction with position and velocity losses in FG-MDM: Towards Zero-Shot Human Motion Generation via ChatGPT-Refined Descriptions to yield natural movements from unconstrained text.
- Dense Correspondences: The benefits extend beyond human bodies; Ao Luo et al. showed in FlowDiffuser: Advancing Optical Flow Estimation with Diffusion Models that predicting the actual flow field instead of noise preserves spatial consistency and long-range dependencies in dense correspondence maps.
Preserving Continuity and Preventing Drift in Sequential Data
For continuous temporal data—such as audio, speech, and multivariate time series—noise prediction introduces a unique vulnerability: the model often confuses the stochastic noise added during the diffusion process with the complex, non-linear variations inherent to the data itself.
Xinyu Yuan et al. addressed this in Diffusion-TS: Interpretable Diffusion for General Time Series Generation, noting that direct signal estimation is far more effective at preserving the smooth, continuous nature of temporal signals. When a model predicts noise, small errors accumulate across iterative steps, leading to severe drifting or physical implausibility in long sequences. Direct signal prediction anchors each step to a coherent global state.
For instance, in long-sequence skeletal generation for music conducting, Zhuoran Zhao et al. found in Taming Diffusion Models for Music-Driven Conducting Motion Generation that direct pose prediction drastically reduces drifting compared to standard noise prediction.
This stability is crucial for maintaining expressive variety and preventing mode collapse. Steven Hogue demonstrated in Joint Co-Speech Gesture and Expressive Talking Face Generation Using Diffusion with Adapters that direct sample prediction prevents complex motion sequences from collapsing into static or averaged poses, resulting in smoother transitions and more dynamic movements.
Similarly, in sign language animation, Lu Dong et al. utilized direct prediction in Word-Conditioned 3D American Sign Language Motion Generation to enhance the structural coherence of synthesized 3D sequences. The same principle underpins the work of Sicheng Yang et al. in DiffuseStyleGesture: Stylized Audio-Driven Co-Speech Gesture Generation with Diffusion Models and its successor The DiffuseStyleGesture+ entry to the GENEA Challenge 2023, where direct reconstruction maintained temporal stability across extended co-speech gestures.
Numerical Stability in Few-Step Inference and Latent Spaces
When diffusion models are restricted to very few sampling steps—a necessity for real-time applications like speech enhancement—noise prediction becomes highly volatile. Small errors in estimating the noise vector at high-noise states can completely derail the trajectory, leading to severe reconstruction artifacts.
Direct signal prediction provides a much safer numerical trajectory. Lei Yue et al. demonstrated in Shallow Diffusion for Fast Speech Enhancement (Student Abstract) that estimating the clean target directly in x-space provides greater numerical stability and fewer artifacts when restricting the sampling depth. This property is echoed by Wenxin Tai et al. in DOSE: Diffusion Dropout with Adaptive Prior for Speech Enhancement, who found that direct prediction prevents "condition collapse" and stabilizes inference when starting from partially informed priors or using minimal timesteps.
Trajectory Stability in Few-Step Inference:
Noise Prediction:
[Pure Noise] ──(high variance ε error)──> [Distorted Output]
Direct Signal Prediction:
[Pure Noise] ──(anchored to x0 manifold)──> [Clean Output]
In non-autoregressive time series forecasting, Chunyan Miao et al. showed in Latent Diffusion Transformer for Probabilistic Time Series Forecasting that predicting target data directly in a compressed latent space outperforms noise prediction, as it prevents the model from conflating the data's inherent stochasticity with the diffusion process's noise.
Facilitating Distillation and Low-Data Personalization
Direct signal prediction also alters the optimization landscape for model compression and fine-tuning. In diffusion model distillation, the goal is to compress a multi-step teacher model into a fast, single-step student model.
Jiatao Gu et al. explored this in BOOT: Data-free Distillation of Denoising Diffusion Models with Bootstrapping and Data-free Distillation of Diffusion Models with Bootstrapping. By reformulating the diffusion process to focus on the evolution of the underlying low-frequency signal component rather than noisy pixel values, the authors aligned the student's predictions with the teacher's boundary conditions. This prevents the student from accumulating out-of-distribution errors when attempting to reconstruct clean images from high-noise states.
Furthermore, direct prediction proves highly advantageous when data is scarce or low-dimensional:
- Low-Dimensional Embeddings: For few-shot learning, Yingjun Du et al. developed ProtoDiff: Learning to Learn Prototypical Networks by Task-Guided Diffusion, showing that direct signal estimation yields significantly more accurate class representations within low-dimensional prototype spaces.
- Sample-Efficient Personalization: In the realm of personalized generative modeling, Balamurugan Thambiraja et al. demonstrated in 3DiFACE: Synthesizing and Editing Holistic 3D Facial Animation that direct signal prediction enhances training stability when fine-tuning models on extremely limited data (such as one-minute reference clips), successfully capturing fine-grained, person-specific stylistic nuances.
Key Foundational Work
Non-autoregressive Conditional Diffusion Models for Time Series Prediction (2023) — contributed 3 concept elements to this direction.
DiffuseStyleGesture: Stylized Audio-Driven Co-Speech Gesture Generation with Diffusion Models (2023) — contributed 2 concept elements to this direction.
Diffusion-TS: Interpretable Diffusion for General Time Series Generation (2024) — contributed 2 concept elements to this direction.