Diffusion Noise as a Tunable Bottleneck for Synthetic-to-Real Geometry

Models that estimate 3D geometry are cheapest to train on synthetic renders, because depth, surface normals and object pose come for free. They are costly to deploy, because the renders are too clean. Networks learn to rely on idealized shading, perfect edges and simplified textures, and they break on real photographs. Latent diffusion models are image generators that compress each picture with an autoencoder and then learn to turn random noise into those compressed codes. They offer an unusual tool for this problem. Adding noise to a compressed image code is exactly the first step of the generator's training process, and the amount of noise sets how much information about the original image survives. We propose to treat that noise level as a dial on an information bottleneck. The idea is to push a synthetic image into the generator's code space, noise it just enough to drown the rendering-specific detail, and let the generator's denoiser, which was trained on billions of real images, rebuild features that look like they came from a real photo. A small geometry head trained on those features from synthetic data alone should then transfer to real images.

The claim is testable and falsifiable, and it cuts through a tension in the literature. One camp treats the autoencoder's compression as a fidelity ceiling to be bypassed with pixel-space models. The other finds that uncompressed, high-dimensional features are exactly what hands domain discriminators their shortcuts. We do not argue for either side. We propose to measure, for 3D geometry, where on the information-rate axis synthetic-only training stops hurting and starts losing geometry.

Core Claim

Consider an image $x$ with compressed code $z = E(x)$ from the frozen autoencoder encoder $E$. The diffusion forward process produces

$$z_t = \sqrt{\bar\alpha_t}\, z + \sqrt{1-\bar\alpha_t}\,\epsilon, \qquad \epsilon \sim \mathcal{N}(0, I),$$

where $\bar\alpha_t \in (0,1]$ is the fraction of signal kept at timestep $t$. This is a Gaussian channel. The standard capacity bound therefore limits the information each code dimension carries about the image:

$$I(x; z_t) \le \sum_{i} \tfrac{1}{2}\log\!\Big(1 + \tfrac{\bar\alpha_t\, \sigma_i^2}{1-\bar\alpha_t}\Big),$$

where $\sigma_i^2$ is the variance of code dimension $i$. The rate of the bottleneck is therefore known and set by $t$. It is not learned, as in variational bottlenecks.

The claim has two parts. First, the synthetic-to-real gap of a geometry head reading denoiser features at timestep $t$ follows a U-shaped curve in $t$. Low $t$ keeps rendering shortcuts, and high $t$ destroys geometry. Second, at the best $t$, the denoiser's reconstruction contributes transfer beyond what information removal alone provides. The ablations isolate this second part. The two-stage structure (compress by the frozen autoencoder, then noise) also lets us separate the autoencoder's fixed low-pass effect from the tunable noise bottleneck.

Relation to Prior Work

Two lines of work motivate the claim. SIBAN (Significance-aware Information Bottleneck for Domain Adaptive Semantic Segmentation) showed that high-dimensional synthetic features carry style and texture cues that let a domain discriminator overpower the segmenter. It fixed this with a learned variational bottleneck plus per-channel relaxation for rare classes. We borrow the diagnosis and the per-channel relaxation. We replace the learned, target-data-dependent bottleneck with the generator's own noise channel, which needs no real images during training. Work on the autoencoder as a low-pass filter (Harnessing Frequency Spectrum Insights for Image Copyright Protection Against Diffusion Models) and on its destruction of fine glyphs (DiffUTE: Universal Text Editing Diffusion Model) suggests that the compression step already erases some high-frequency rendering signatures. That is a hypothesis we test rather than assume.

Two papers supply the method. DMP (Exploiting Diffusion Prior for Generalizable Dense Prediction) adapts a frozen Stable Diffusion with low-rank adapters on synthetic bedroom images and transfers depth and normals to stylized images. DAG (Depth-aware guidance with self-estimated depth representations of diffusion models) reads depth from internal denoiser features with small MLPs trained on about 100 images. DMP is the closest existing work. It already trains on synthetic data inside a diffusion model, but it treats the timestep as an implementation detail and evaluates on stylized art rather than on real photographs against a synthetic-only reference. We make the timestep the object of study, characterize the transfer curve, add channel-wise significance gating, and test on real imagery.

GAN2Shape (Do 2D GANs Know 3D Shape? Unsupervised 3D Shape Reconstruction from 2D Image GANs) motivates the "project onto the real-image manifold" intuition. It showed that projecting distorted shape hypotheses back onto a GAN's image manifold corrects them. We apply the same projection to synthetic inputs rather than shape hypotheses.

Other work constrains the design. Bousmalis et al. (Unsupervised Pixel-Level Domain Adaptation with Generative Adversarial Networks) state that pixel-level translation handles only low-level appearance gaps, so we do not expect our method to fix genuine shape mismatches between CAD models and real objects either. We evaluate on tasks where the gap is mostly appearance. Pixel-space advocates, HDiT (Scalable High-Resolution Pixel-Space Image Synthesis with Hourglass Diffusion Transformers) and GenRec (GenRec: Knowing Where to Reconstruct and Where to Generate), appear to contradict us. Both show that compression costs fine detail. We take this seriously. Thin structures and depth discontinuities are exactly what a strong bottleneck should hurt, so we report boundary-specific error separately and expect a real fidelity cost at high $t$. The sim-to-real LiDAR results of Li et al. (Adversarially Masking Synthetic to Mimic Real: Adaptive Noise Injection for Point Cloud Segmentation Adaptation) show that injecting noise into clean synthetic input removes a shortcut. Our work can be read as moving that injection into a learned generative code space, where it can be followed by a realistic reconstruction.

Research Program

We build one pipeline around a frozen Stable Diffusion model. A synthetic image is encoded, noised to timestep $t$, and passed once through the frozen denoiser. Multi-scale decoder features are read out, and a lightweight head predicts depth and normals, or object pose. The head is trained only on synthetic labels. An optional variant adds a SIBAN-style $1\times1$ gating layer, which lets channels that carry rare-but-needed geometric cues pass at a lower effective noise level.

The first study sweeps $t$ densely and traces the synthetic-validation error and real-test error as functions of the capacity bound above. This yields a gap curve. The second study decomposes the effect into three contributions: what the autoencoder alone removes (features from $z$ with no noise and no denoiser), what noise alone removes (noised codes fed to a head without the denoiser), and what the denoiser adds on top.

The third study asks whether $t$ can be chosen without real labels. We train a small domain classifier to separate synthetic from unlabeled real features at each $t$. We then test whether the $t$ at which its accuracy drops toward chance, subject to synthetic validation error staying within a fixed tolerance, matches the $t$ that minimizes real error. If it does, the method is fully unsupervised on the target side.

Baselines

The naive reference is a standard monocular depth or pose network of comparable parameter count, trained on the same synthetic images and tested on real ones. It shows how large the gap is before any intervention.

The strongest existing method for dense geometry is DMP (Exploiting Diffusion Prior for Generalizable Dense Prediction), run with its published recipe of low-rank adapters, deterministic interpolation and five-step inference. It uses the same synthetic training set and the same Stable Diffusion checkpoint, and receives the same number of training steps and hyperparameter trials as our head. DMP reports RMSE 0.1360 on arbitrary stylized images versus 0.2554 for SPADE, and we will reproduce that number before comparing. For real indoor photographs such as NYU-Depth, we found no reported result on Lacuna for DMP trained on synthetic data. We will run it ourselves. Beating it means lower real-image depth error and normal angular error at matched inference cost, with no loss on the stylized set.

For object pose, the strongest competitor with a reported number is PixelDA (Unsupervised Pixel-Level Domain Adaptation with Generative Adversarial Networks). PixelDA reports a mean angle error of 23.5 degrees on Synthetic-to-Real LineMod, against 56.58 for DANN. We will compare against 23.5 degrees directly, using the same pose-head architecture PixelDA used where feasible. Beating it means a lower mean angle error with no real target images used in training, whereas PixelDA uses unlabeled target images. We will also report a variant that uses unlabeled target images only for selecting $t$.

A frequency-control baseline applies Gaussian blur, or additive pixel noise, to synthetic images before training the naive network. Its strength is swept so that its estimated information rate matches our chosen $t$.

Finally, a no-denoiser ablation reads noised autoencoder codes directly. This isolates whether the generative prior's reconstruction, rather than information removal, drives the result.

Why not just…?

Why not just blur or add noise to the synthetic images? This is the most serious threat. If the gap is simply high-frequency rendering artifacts, a pixel-space low-pass filter at matched rate could do as well, and the diffusion model would be an expensive blur. Nothing in the file rules this out. The matched-rate blur and noise baselines are designed to settle it. If they close most of the gap our method closes, we will report that the benefit is information removal, not the generative prior, and that is a useful finding in its own right. Our reason to expect otherwise is that blur removes detail uniformly, including depth edges, whereas the denoiser re-synthesizes plausible real-image detail. The boundary-error metric should expose this difference.

Why not just use DMP as published? DMP may already sit near the best timestep by accident. The sweep will show this. If DMP's default timestep is at the minimum of our curve and gating adds nothing, the contribution shrinks to an explanation of why DMP transfers, plus a label-free selection rule. We would state that plainly rather than claim a new method.

Why not just translate synthetic images to look real with a GAN? Pixel translation needs unlabeled target data and adversarial training, and its authors restrict it to low-level appearance gaps (Unsupervised Pixel-Level Domain Adaptation with Generative Adversarial Networks). Our method shares that limit on geometry, so the LineMod comparison against the reported 23.5 degrees is a direct test of whether a frozen web-scale prior matches a target-specific translator without seeing target data.

Why not just supervise real images with an off-the-shelf depth teacher? Teachers inherit their own training domain. Lee et al. show that standard predictors degrade on non-photorealistic inputs, and GenRec reports artifacts whenever its monocular teacher errs (GenRec: Knowing Where to Reconstruct and Where to Generate). Teachers also give no pose labels. We include teacher-pseudo-labeled real training as an upper reference for depth, but not as a competitor on the synthetic-only question.

Proposed Experiments

Dense geometry is trained on synthetic indoor renders, following DMP's synthetic bedroom setup, and tested on real NYU-Depth images and on DMP's stylized evaluation set. We report RMSE, threshold accuracy at $\delta < 1.25$, normal angular error, and error restricted to pixels within a few pixels of depth discontinuities. Object pose is trained on Synthetic LineMod renders and tested on real LineMod, reporting mean angle error and classification accuracy. As a stress test for the per-channel gating, we run GTA5 to Cityscapes segmentation, where DMP noted failure on many-class segmentation and SIBAN reports 42.6% mIoU (Significance-aware Information Bottleneck for Domain Adaptive Semantic Segmentation). We do not expect to beat SIBAN there. We will use it to check whether gating prevents rare classes from being erased at high noise.

Three results would be decisive. The first is a clear interior minimum of real-image error as a function of $t$, while synthetic error rises steadily. The second is that the full denoiser pipeline beats both the no-denoiser ablation and the matched-rate blur and noise baselines on real error at that minimum. The third is that the label-free, domain-classifier-based choice of $t$ lands within one sweep step of the best $t$. If the first holds but the second fails, the paper becomes a measurement study of where the synthetic-to-real gap lives in frequency. If both hold, we have a training recipe that needs no real images and a principled reason it works.