Diffusion models typically begin their generative journey from a state of pure chaos: a grid of independent, identically distributed (i.i.d.) Gaussian noise. While this works for 2D images, it creates a fundamental "consistency crisis" when applied to 3D scenes. If two different camera views start with two different random noise patterns, the diffusion model will attempt to generate two different visual realities for the same physical space.
To solve this, researchers have moved toward structured noise initialization. By anchoring the initial stochastic state to the underlying 3D geometry, models can ensure that the generative process is physically consistent across viewpoints, preventing artifacts like "texture sticking," flickering, and the infamous "Janus face" problem.
Anchoring Noise to Surfaces
The most direct way to ensure consistency is to make the noise a property of the 3D surface itself rather than the 2D image plane. In ConsistDreamer, Chen et al. propose generating noise directly on the 3D geometry of a scene and then projecting it into 2D camera views. This ensures that if a specific point on a chair is visible from two different angles, both views will "see" the same initial noise values at that location.
Similarly, Ye demonstrates in DiffPBR that assigning fixed noise vectors to 3D points in a cloud and splatting them into 2D space creates a spatially consistent noise map. This mechanism forces the diffusion process to remain stable across temporal frames and viewpoints, effectively eliminating the flickering common in neural rendering.
Global Noise Fields and Warping
When a full 3D mesh isn't available, researchers use geometric transformations to synchronize noise. Hu et al. and the authors of In-N-Out align initial latents by warping noise from a reference view into target views using the scene's depth. This "outstretches" a single-view generative seed into a coherent 3D structure. Müller et al. apply a similar logic in MultiDiff, showing that the statistical correlation of the initial noise distribution is a primary driver of multi-view coherence.
For more dynamic optimization, Yan et al. introduce a "world-map" noise function in Flow Score Distillation. Instead of sampling new noise at every step, the model queries a global, static noise distribution based on spatial coordinates. This prevents the "diversity crisis" where models produce identical results regardless of the seed, while ensuring that the noise added to rendered images remains spatially consistent as the camera moves.
Geometric Proxies as Structural Seeds
Another facet of this research direction replaces pure Gaussian noise with "noisy proxies"—low-fidelity geometric guides that provide a structural head start.
- Coarse Shapes: Li et al. use Gaussian blobs as structural starting points in Instant3D to ensure objects are centered and isolated.
- 3D Priors: Gu leverages noise patterns derived from pre-trained 3D-aware generators in DiffPortrait3D and DiffPortrait360. This provides a "geometric skeleton" that allows the diffusion model to focus on high-fidelity texture without having to learn basic 3D structure from scratch.
- Test-Time Control: Fedele shows in SpaceControl that initializing with a noisy latent representation of a 3D sketch allows for precise spatial control without retraining the model.
By starting the denoising process at a point where the coarse geometry is already defined, these methods bypass the high-uncertainty early stages of diffusion where structural hallucinations (like the Janus problem) typically take root (Liu et al.; Yang et al.).
Topology-Aware Noise Distributions
Finally, the structure of the noise can be tailored to the specific topology of the data. In Spherical-Nested Diffusion, Sun et al. argue that standard 2D noise is ill-suited for panoramic images. By sampling noise on a 3D sphere before projecting it to a 2D equirectangular format, the model becomes inherently aware of polar distortions.
This geometric awareness extends to other domains as well:
- Medical Imaging: Du et al. use spherical interpolation to synchronize noise across 2D slices, creating a consistent 3D volume in ISCS.
- Molecular Modeling: Aykent et al. utilize "direction noise" on a sphere to accelerate the training of geometric vector networks in SaVeNet.
Across all these applications, the core intuition remains the same: for a generative model to respect 3D space, its very first "thought"—the initial noise—must be grounded in that space.
Key Foundational Work
ConsistDreamer: 3D-Consistent 2D Diffusion for High-Fidelity Scene Editing (2024) — contributed 2 concept elements to this direction.
Flow Score Distillation for Diverse Text-to-3D Generation (2024) — contributed 2 concept elements to this direction.
LucidDreamer: Towards High-Fidelity Text-to-3D Generation via Interval Score Matching (2024) — contributed 2 concept elements to this direction.