Diffusion models turn random noise into images by running a learned denoising procedure over many small steps. The same input can give an excellent result under one noise seed and a hallucinated one under another. In super-resolution, seed alone can swing reconstruction quality by more than 4 dB in PSNR. Separately, the largest image generators are increasingly built as mixtures of experts, in which each image patch ("token") is sent to only a few of many parallel sub-networks by a small routing network. These two facts have not been connected. A routing decision is discrete: a token either goes to an expert or it does not. So the function the sampler integrates jumps whenever a token crosses a routing boundary. A tiny change in the starting noise can therefore flip which experts a token uses at some step, and the rest of the trajectory can diverge.
This note proposes to measure whether sparse routing makes diffusion samplers more seed-sensitive than dense models of equal quality, and to test a cheap remedy. The remedy blends the two nearest routing choices only for tokens that sit close to a routing boundary. This makes the denoising function continuous again at a small fraction of the cost of fully soft routing. If routing flips do amplify the seed lottery, every mixture-of-experts image model now being scaled carries a hidden reliability cost. That cost would matter most for super-resolution, inverse problems and video, where users need the same answer every time.
Core Claim
With a fixed-step ODE solver, the map from starting noise to final image is a composition of per-step updates. In a dense network each update is smooth. In a sparse mixture of experts, each update is only piecewise smooth, with jumps on the surfaces where a token's routing score crosses a threshold or a top-$k$ rank changes. The claim has two parts. First, near these surfaces the seed-to-output map has jumps rather than merely steep slopes, and those jumps show up as measurably higher local and global seed sensitivity in sparse models than in dense models at matched sample quality. Second, removing the jumps for the small set of near-boundary tokens recovers dense-level stability without giving up the quality gains of sparse routing.
We quantify local sensitivity as
$$S(\varepsilon) = \mathbb{E}_{z,u}\left[\frac{d\big(G(z),\, G(\mathrm{norm}(z+\varepsilon u))\big)}{\varepsilon}\right],$$
where $G$ is the full sampler, $z$ a starting noise, $u$ a random unit direction, $\mathrm{norm}$ rescales back to the Gaussian shell, $\varepsilon$ the perturbation size, and $d$ a distance such as LPIPS or $L_2$. For a smooth map, $S(\varepsilon)$ approaches a constant as $\varepsilon \to 0$. For a map with jumps, the jumps contribute a term that falls roughly like $1/\varepsilon$, because a fixed-size jump is divided by a shrinking perturbation. The shape of $S(\varepsilon)$ is therefore a direct test of the hypothesis, separate from any quality metric.
The remedy, which we call margin blending, keeps routing sparse everywhere except inside a band of width $\delta$ around each decision boundary. Consider a threshold-style router such as the identity gating in Expert Race, which sends a token to an expert when its routing score $s$ exceeds a threshold $\tau$. Margin blending replaces the hard on/off decision with a weight
$$w(s) = \mathrm{clip}\!\left(\frac{s-\tau+\delta}{2\delta},\,0,\,1\right),$$
so the expert contributes $w(s)$ times its output. For top-$k$ routers, the same interpolation is applied between the $k$-th and $(k{+}1)$-th ranked experts when their scores are within $\delta$. The resulting denoising function is continuous in the input. Extra compute is incurred only for tokens inside the band, and we will report that fraction directly. The method can be applied after training to an existing checkpoint, or used during a short fine-tune.
Relation to Prior Work
The motivation comes from the stochasticity literature, which established that the starting noise acts as a boundary condition with large downstream effects. PNFS documents seed-induced PSNR swings of more than 4 dB in super-resolution and fixes them by predicting per-pixel errors for several candidate noises and fusing them. ANSE uses first-step attention entropy to discard bad seeds in video models. Optimal boundary conditions precomputes one good starting noise offline. All three treat the network as a fixed smooth map and act on its input. We instead ask whether the network's own architecture enlarges the problem, a question these methods cannot answer because they never vary the architecture.
The architectures under study come from three sparse diffusion transformer papers. DiT-MoE uses top-$k$ routing plus shared experts and reports that routing is concentrated at high noise and diffuse at low noise. Race-DiT uses identity-gated thresholds tracked by an exponential moving average, so each token gets a variable number of experts. ProMoE uses prototype cosine-similarity routing and a hard split between conditional and unconditional passes. All three evaluate only FID and prompt alignment, and none measures run-to-run or seed-perturbation variance.
DiT-MoE's finding that late-step routing is diffuse suggests a specific risk: low-noise, texture-level tokens hover near boundaries. That is exactly where super-resolution hallucinations appear. SS-MoE routes on the timestep alone. Its routing is therefore trivially continuous in the image, and it becomes an important baseline rather than prior art on our question. Geometric and Stochastic Analysis of Discontinuities in Sparse Mixture-of-Experts analyzes the jumps that sparse selection introduces. We take that as theoretical support and measure its consequence for the seed-to-output map of an iterated generative sampler, which, as summarized in the research file, it does not address.
Two cross-direction papers are not competitors on this question. VDD removes diffusion stochasticity by distilling a low-dimensional robot policy into a Gaussian mixture, which does not transfer to images. CATImage routes whole prompts across separate models and explicitly names uncontrolled seed stochasticity as an unresolved obstacle.
Research Program
The work has three stages that share one set of models. In the diagnostic stage we train matched model families on ImageNet 256×256 at base and XL scale: a dense DiT, a top-$k$ model with shared experts in the DiT-MoE style, an Expert Race model, a ProMoE-style prototype router, and a timestep-only SS-MoE-style variant. All sparse models have the same number of active parameters. For each model we measure $S(\varepsilon)$ over five orders of magnitude of $\varepsilon$ using a fixed 50-step deterministic solver, and we also record the routing flip rate, meaning the fraction of (token, layer, step) decisions that change under the perturbation. The key analysis relates output divergence to where and when the first flip occurs. If the hypothesis is right, divergence should be concentrated in trajectories whose first flip happens early and in shallow layers.
The remedy stage applies margin blending post hoc at several values of $\delta$ and after a short fine-tune. We track sensitivity, FID-50K and the extra compute as $\delta$ varies.
The downstream stage uses each class-conditional prior as the generative prior for 4× super-resolution and Gaussian deblurring on ImageNet-256. All priors use the same posterior sampler; we use the derivative-free path-space resampling of URGE as one sampler and a single-particle guided sampler as the other. For each degraded image we draw 16 seeds and report the PSNR spread (worst-to-best), the per-pixel standard deviation across seeds in the style of PNFS's local standard deviation, and mean LPIPS. We also perturb the degraded input slightly, adding fresh measurement noise, and repeat the analysis. This probes the frame-to-frame instability that matters for video, because neighbouring frames are near-identical inputs.
Baselines
The naive reference is each sparse model sampled with plain Gaussian seeds and hard routing. It anchors both the quality and variance axes. The dense DiT at matched active parameters is the second reference; it shows how much variance is attributable to sparsity rather than to diffusion itself. For generation quality, the strongest existing reference on this benchmark is Race-DiT-XL, which reports FID 3.31 on ImageNet 256×256 against 6.31 for its dense XL counterpart (Expert Race). ProMoE-L-Flow reports FID 2.79 on ImageNet (ProMoE). For seed variance of any sparse diffusion model there is no reported result found on Lacuna, so our own dense and hard-routed measurements define the reference points.
For stability, the strongest existing method is PNFS. It reports up to a 4.42 dB PSNR uplift in worst-case seeds, a local standard deviation of about 0.02, and 0.03 seconds of added inference time (PNFS). We will run PNFS's noise-fusion procedure on top of every prior, including the dense one, with the same number of candidate noises and a discrepancy predictor retrained on the same DIV2K/Flickr2K split, so that it acts on identical backbones. We also include ANSE-style first-step seed filtering with the same candidate pool size, and a single precomputed optimal starting noise in the style of optimal boundary conditions.
Several ablations isolate the key ingredient. The SS-MoE-style timestep router tests whether continuity alone, without token adaptivity, is enough. Fully soft routing over all experts, CondConv-style (CondConv), gives the continuity ceiling at full compute. A random-token blending control spends the same extra compute as margin blending but on tokens chosen without regard to their distance from a boundary. If random blending helps as much as margin blending, the boundary explanation is wrong.
All models share data, solver, step count, total training tokens and the tuning budget for $\delta$ and fine-tuning. Beating the field means two things. At matched active compute plus at most a few percent extra, margin blending must bring $S(\varepsilon)$ and the 16-seed PSNR spread of a sparse model to within noise of the dense model, while keeping FID within 0.1 of its hard-routed version. It must also be additive with PNFS, meaning that PNFS on a margin-blended prior shows a smaller worst-case spread than PNFS on the hard-routed prior.
Why not just…?
Why not just use a dense model if stability matters? Because the sparse models are much better on quality at the same active compute: FID 3.31 versus 6.31 in the Race-DiT comparison. Giving that up for reliability is precisely the trade this work tries to avoid. The dense model stays in every table. If sparse models turn out to be no more seed-sensitive than dense ones, that null result is itself useful and the remedy is unnecessary.
Why not just fix the seed, or use a deterministic solver? A fixed seed gives exact reproducibility but not robustness. Two nearly identical inputs, such as adjacent video frames or the same photo with different sensor noise, still travel through different routing patterns. The input-perturbation experiment measures exactly this case. Deterministic solvers are already assumed here: the jumps come from the routing itself, not from injected noise.
Why not just route on the timestep, as SS-MoE does? This is the most serious threat. Timestep-only routing is continuous in the image at zero extra cost. Its weakness is that it gives up per-token allocation, which Race-DiT argues is the main source of sparse-model gains in vision because flat backgrounds and detailed foregrounds need different compute. If the timestep-only variant matches the token-routed models on FID while matching dense stability, we will report that and recommend it over margin blending.
Why not just use PNFS or ANSE on top of any model? These methods choose or fuse starting noises, and ANSE's own authors note that a selector cannot fix a bad pool. More to the point, they multiply the number of candidates evaluated, and a model with a jumpier seed map needs more candidates to find a good one. The additivity test shows whether stabilizing the architecture reduces how many candidates such methods need.
Proposed Experiments and Decisive Checks
The first decisive check is the shape of $S(\varepsilon)$. If the sparse models show the roughly $1/\varepsilon$ rise at small perturbations that the dense and timestep-only models lack, and margin blending removes it, the mechanism is confirmed. If all curves flatten identically, routing jumps are numerically negligible and the project should stop there with a negative result.
The second is the flip-attribution analysis: most large output divergences should be preceded by an early routing flip, and the random-blending control should not reproduce the gain.
The third is downstream. On ImageNet-256 4× super-resolution and deblurring, margin blending must shrink the 16-seed PSNR spread and the per-pixel cross-seed standard deviation of a sparse prior without raising mean LPIPS, and must reduce PNFS's worst case when the two are combined.
Throughout, we report the fraction of tokens inside the blending band per timestep. DiT-MoE's observation that routing is diffuse at low noise predicts that this fraction peaks late in sampling. If it exceeds roughly a tenth of tokens, the method's cost argument fails and fully soft late-step routing becomes the honest recommendation.
References
- Expert Race: A Flexible Routing Strategy for Scaling Diffusion Transformer with Mixture of Experts
- Restabilizing Diffusion Models with Predictive Noise Fusion Strategy for Image Super-Resolution
- Model Already Knows the Best Noise: Bayesian Active Noise Selection via Attention in Video Diffusion Model
- Solving Diffusion ODEs with Optimal Boundary Conditions for Better Image Super-Resolution
- Scaling diffusion transformers to 16 billion parameters
- Routing Matters in MoE: Scaling Diffusion Transformers with Explicit Routing Guidance
- Image Super-Resolution via Latent Diffusion: A Sampling-Space Mixture of Experts and Frequency-Augmented Decoder Approach
- Geometric and Stochastic Analysis of Discontinuities in Sparse Mixture-of-Experts
- Variational Distillation of Diffusion Policies into Mixture of Experts
- Cost-Aware Routing for Efficient Text-To-Image Generation
- Simple Approximation and Derivative Free Inference-Time Scaling for Diffusion Models via Sequential Monte Carlo on Path Measures
- CondConv: Conditionally Parameterized Convolutions for Efficient Inference