Rao-Blackwell Gradient Estimators for Equivariant Denoising Diffusion

Vinh Tong, Hoang Trung-Dung, Anji Liu, Guy Van den Broeck, Mathias Niepert

TL;DR

This paper introduces Orbit Diffusion (OrbDiff), a novel framework that applies Rao-Blackwellization to data augmentation in diffusion models to drastically reduce gradient variance during training. By replacing noisy sample targets with a calculated conditional expectation over group orbits, OrbDiff achieves faster convergence and more stable training for both equivariant and standard architectures—all with virtually zero computational overhead.

What Problem Does This Paper Address?

Modeling physical systems like molecules, crystals, and proteins requires capturing inherent geometric symmetries (such as rotations, translations, and reflections). Currently, researchers use two main strategies to handle this:

  1. Equivariant architectures: Networks designed to mathematically guarantee symmetry, which are often complex, slow, and difficult to optimize.
  2. Data augmentation: A flexible alternative that trains standard networks on randomly transformed inputs, effectively acting as a noisy Monte Carlo estimator of the true symmetric gradient.

However, standard data augmentation suffers from high variance, leading to unstable training and slow convergence. The authors address this fundamental trade-off by asking: Can we retain the flexibility of data augmentation while eliminating its high-variance drawbacks?

Methods

The authors re-interpret data augmentation through a rigorous probabilistic lens and apply Rao-Blackwellization—a classic statistical technique used to reduce the variance of an estimator by conditioning it on a sufficient statistic.

In standard diffusion training, the network tries to predict a clean sample $x_0$ from a noisy version $x_t$. The paper's key insight is that because the neural network's prediction $\phi(x_t, t)$ does not depend on the clean sample $x_0$, the training gradient can be mathematically decomposed. Instead of computing the loss against a single randomly augmented sample, they replace the target $x_0$ with its conditional expectation given the noisy sample, $\mathbb{E}[x_0 | x_t]$.

To make this tractable in practice without exploding computational costs, they introduce Orbit Diffusion (OrbDiff). Instead of averaging over the entire dataset, OrbDiff restricts its focus to the orbit of the specific sample (i.e., all possible symmetric transformations of that single data point). It uses Self-Normalizing Importance Sampling (SNIS) to efficiently approximate this conditional expectation.

As shown in the framework overview below, this entire process requires only a single forward and backward pass per sample, making it incredibly efficient.

Figure 1: Gradient estimation strategies for training (approximately) equivariant diffusion models: (a) Sampling from the symmetrized joint distribution to obtain x0 and xt. (b) The standard data augmentation approach, which directly uses these samples for training. (c) The proposed method, leveraging self-normalizing importance sampling (SNIS) to estimate the inner conditional expectation. Both (b) and (c) require a single neural function evaluation per gradient step, but (c) has lower variance than (b). The pseudo-code for the Rao-Blackwell estimator with SNIS is shown on the right.

Datasets

The authors evaluated OrbDiff across several generative benchmarks:

  • Synthetic 1D Dataset: A controlled setup exhibiting reflection symmetry.
  • GEOM-QM9: A standard molecular dataset used to evaluate molecular conformation generation under 3D rotation and graph automorphism symmetries.
  • Perov-5 & MP-20: Benchmarks for crystal structure prediction, evaluated under both standard and text-guided generation tasks.
  • Foldseek AFDB clusters: Used for large-scale protein structure generation.

Key Insights and Ideas

  • Factoring out the Network: The "aha moment" relies on realizing that the neural network only sees $x_t$. This allows researchers to pull the network out of the inner expectation over the symmetry group, meaning you can average multiple symmetric targets before backpropagation without needing multiple expensive forward passes.
  • Augmentation Doesn't Help Equivariant Models (But OrbDiff Does): The authors formally prove that standard data augmentation provides zero benefit if a model is already structurally equivariant. However, applying OrbDiff to an equivariant model does help because it directly targets and reduces gradient variance.
  • Smarter Sampling Matters: For continuous groups like translations or rotations, uniform sampling can be inefficient at low noise levels. The authors introduced a time-dependent Wrapped Normal proposal (OrbDiff_WN) that samples locally near the identity when noise is low and expands globally as noise increases, significantly boosting performance.

Key Results

OrbDiff consistently outperformed baselines across all domains:

  • Synthetic Data: Achieved a 25× lower root-mean-square deviation (RMSD) and a 200× lower Wasserstein-2 distance compared to standard data augmentation, yielding a significantly smoother loss curve.
  • Molecular Generation (GEOM-QM9): Integrating OrbDiff into ETFLOW achieved state-of-the-art recall scores, including a 4% absolute improvement in mean Coverage ([email protected]).
  • Crystal Structure Prediction: On the text-guided MP-20 benchmark, OrbDiff boosted the structural match rate from 61.91% to 66.74%.
  • Protein Generation: When applied to PROTEINA (a non-equivariant transformer), OrbDiff improved the designability fraction from 93.8% to 95.6%.
  • Efficiency: The computational overhead was negligible—adding only 0.1% extra training time for large models like PROTEINA.

Limitations and Future Work

While highly effective, the authors note that their orbit sampling scheme inherently constrains the choice of proposal distributions, which might limit flexibility for tasks requiring highly adaptive or diverse noise structures. Additionally, the number of orbit samples chosen ($N$) creates a direct trade-off between computational cost and estimation accuracy. The authors suggest that dynamically adjusting the sampling strategy and sample size throughout the training process is a promising direction for future work.

Summary updated

Abstract

In domains such as molecular and protein generation, physical systems exhibit inherent symmetries that are critical to model. Two main strategies have emerged for learning invariant distributions: designing equivariant network architectures and using data augmentation to approximate equivariance. While equivariant architectures preserve symmetry by design, they often involve greater complexity and pose optimization challenges. Data augmentation, on the other hand, offers flexibility but may fall short in fully capturing symmetries. Our framework enhances both approaches by reducing training variance and providing a provably lower-variance gradient estimator. We achieve this by interpreting data augmentation as a Monte Carlo estimator of the training gradient and applying Rao-Blackwellization. This leads to more stable optimization, faster convergence, and reduced variance, all while requiring only a single forward and backward pass per sample. We also present a practical implementation of this estimator incorporating the loss and sampling procedure through a method we call Orbit Diffusion. Theoretically, we guarantee that our loss admits equivariant minimizers. Empirically, Orbit Diffusion achieves state-of-the-art results on GEOM-QM9 for molecular conformation generation, improves crystal structure prediction, and advances text-guided crystal generation on the Perov-5 and MP-20 benchmarks. Additionally, it enhances protein designability in protein structure generation. Code is available at: https://github.com/vinhsuhi/Orbit-Diffusion.git.
BibTeX
@inproceedings{tong2025rao,
  author = {Vinh Tong and Hoang Trung-Dung and Anji Liu and Guy Van den Broeck and Mathias Niepert},
  title = {{Rao-Blackwell Gradient Estimators for Equivariant Denoising Diffusion}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2025},
  url = {https://openreview.net/forum?id=ENJhSqVkXH},
}
Figure