Object-Coherent Noise for Diffusion Depth on Small, Distant Objects

Diffusion models are generators that learn to turn random noise into samples by removing it step by step. They now estimate depth from a single image: the model starts from random noise shaped like a depth map and denoises it while looking at the photo. That starting noise is drawn independently for every pixel. We argue this choice matters most where monocular depth is hardest. A distant car or pedestrian covers a few pixels, and the image alone cannot tell whether it is small and near or large and far. With independent noise, the pixels of that object start from unrelated random values, and the denoiser may pull them toward different depths. The result is an object that is torn between foreground and background, or that bleeds into the road behind it. The proposal is to draw noise that is shared within each object region while keeping each pixel's noise standard Gaussian. Each sample from the model then commits to one depth hypothesis per object, and different samples explore different hypotheses along the camera ray.

The question is concrete and testable. The intended outcome is better depth on small objects and better-calibrated uncertainty about where they sit along the ray, with no change to the network. A second ingredient is about resolution. Diffusion depth models work on a downsampled latent grid, so a sub-8-pixel object can be averaged out of the noise field before denoising even begins. We build the region-coherent noise at full image resolution. We then carry it into the latent grid with importance-weighted pooling borrowed from small-object detection, so tiny regions keep their own noise instead of being absorbed into the background.

Core Claim

Independent per-pixel initial noise causes two failures that the field has so far treated as separate. One is incoherent depth within small objects, a resolution problem. The other is incoherent sample diversity under monocular ambiguity, a posterior problem. Noise that is shared within image regions and pooled into the latent grid in a way that preserves small regions should fix both. It should beat simpler alternatives that only change smoothness, resolution, or the starting point of the reverse process.

Concretely, for full-resolution pixel $p$ belonging to region $r(p)$, the initial noise is

$$ \epsilon(p) = \sqrt{1-\rho}\,\xi(p) + \sqrt{\rho}\,\eta_{r(p)}, $$

where $\xi(p)$ and $\eta_r$ are independent standard Gaussians (per pixel and per region respectively) and $\rho \in [0,1]$ sets how strongly a region shares its noise. Every pixel's marginal is still $\mathcal{N}(0,1)$; only the joint correlation changes. The field is then pooled to the latent resolution with learned importance weights and rescaled to unit variance per latent cell. The rescaling is needed because averaging Gaussian noise shrinks its variance, and the denoiser expects unit-variance input.

Relation to Prior Work

Three works motivate the idea and supply its parts. DiffusionDepth turns monocular depth into latent denoising guided by Swin Transformer and FPN features, and it handles KITTI's roughly 4% lidar coverage through self-diffusion. It starts from uncorrelated Gaussian noise. We keep its model and change only the starting noise, plus a short fine-tune. ISCS correlates noise across CT slices by spherical interpolation between two anchor noises. It shows that structured noise keeps small lesions that Total Variation smoothing erases, and that identical noise across slices causes copying artifacts. That second finding constrains our design: $\rho=1$ is the analogue of their failure case, so we treat $\rho$ as a quantity to sweep, not to max out. ISCS correlates noise along a fixed slice axis. We correlate it within image regions of a single view, where no such axis exists. LIP replaces average pooling with pooling weighted by a learned per-pixel importance map and raises small-object AP on COCO from 21.9% to 24.0% with Faster R-CNN. We reuse its pooling to downsample the noise field rather than features.

Two works sit closest and set the novelty boundary. PVDepth samples noise on the sphere for panoramic depth. It anchors noise to known camera geometry and needs an annealing schedule so that the pretrained backbone is not destabilized. We anchor to image content instead of camera geometry, and we expect to need a similar mixed-noise fine-tune. D³-RSMDE blends a ViT's coarse depth into intermediate diffusion latents and reports 40× speedup and 11.85% lower LPIPS than Marigold. It structures the reverse process with a deterministic prior, not with the noise distribution. It therefore yields one answer per image rather than coherent alternatives, and it was not evaluated on tiny targets. Point-anchored noise in DiffPBR requires a known point cloud, which is exactly what monocular depth lacks. ObjectFusion argues that dense pixel depth is the weak link when lifting to 3D. That supports our focus on making per-object depth coherent, but its fix depends on LiDAR, and we do not use LiDAR.

Research Program

We build three things. The first is a region map computed at full resolution from the RGB image. The default is class-agnostic instance masks plus superpixels for background, so that every pixel belongs to some region. A variant uses regions defined by the importance map itself. The second is the noise sampler above, with importance-weighted pooling into DiffusionDepth's latent grid. The importance logits come from a 1×1 LIP logit module applied to the image features, so the noise pooling tracks the same evidence the denoiser sees. The third is a training protocol. We first test the sampler training-free. We then fine-tune DiffusionDepth for a short schedule in which each batch draws either independent or region-coherent noise, with the mixing ratio annealed in the manner of PVDepth.

The analysis has two parts. The first is a stratified error study on KITTI by object pixel area. We expect the gain from coherent noise to grow as objects shrink, and to vanish or reverse for large objects and textured background, where sharing noise could over-regularize. The second is a study of ambiguity. We draw ten samples per image and measure, for each object, how depth varies within the object inside one sample, which captures fragmentation. We also measure how depth varies across samples, which captures hypothesis spread, and whether that spread predicts the object's actual error. Under monocular ambiguity, a good sampler should vary object depth across samples and keep each object intact within a sample. Independent noise is predicted to do the opposite for small objects.

Baselines

The naive reference is DiffusionDepth's deterministic regression counterpart: the same backbone and decoder trained to regress depth directly. It tells us how much any diffusion variant buys on small objects at all.

The strongest existing method is DiffusionDepth with standard independent noise. It reports RMSE 1.418 on the KITTI official split and 0.295 on NYU-Depth-V2. We rerun it from the released configuration with the same step count (20), backbone, and data. All our variants use the same number of fine-tuning iterations as a matched "continue training with independent noise" control, so that extra training is not mistaken for a noise effect. Beating it means one of two things. Either small-object RMSE drops by more than the seed-to-seed spread, with full-image KITTI RMSE no worse than the rerun. Or the fragmentation and calibration measures improve while accuracy stays equal. No reported result was found on Lacuna for small-object-stratified depth error on KITTI, so we produce that number ourselves for every method.

The prior-structured competitor reimplements the D³-RSMDE recipe on top of DiffusionDepth. The regression output is blended into intermediate latents, with the same total step budget. It tests whether giving the reverse process a structured starting point is enough without structuring the noise.

Three ablations isolate the key ingredient. The first uses low-frequency correlated noise with matched $\rho$ and matched average correlation length, but with regions that ignore image content (random blobs). It separates "correlated noise" from "noise anchored to objects." The second replaces importance-weighted pooling with plain average pooling plus rescaling. It isolates the resolution-preserving step. The third sweeps $\rho$ from 0 to 1, with the endpoint reproducing the identical-noise failure that ISCS reports.

Why not just…?

Why not just start from a coarse prediction, as D³-RSMDE does? This is the most serious alternative, because it also stops the denoiser from inventing macro-geometry from scratch. Its weakness for our question is that a deterministic prior sees the same blurred features as the regressor. D³-RSMDE itself describes ViT predictions as low-pass. So the prior is likely already wrong on a tiny object, and the refiner starts from that error. It also removes most of the sample diversity. The prior-structured baseline above settles this. If it matches our small-object error and calibration, the region-coherent noise adds nothing, and we would report that.

Why not just filter each independent-noise sample by taking the median depth within each segment? Post-hoc smoothing within segments also yields coherent object depth and costs almost nothing, so it is a real threat on accuracy. ISCS found that smoothing applied after the fact (Total Variation) erases small structures that noise-level coupling keeps. But that result is for CT slices, not image segments. We run segment-median filtering on the independent-noise samples with the same region maps. Our hypothesis is that once a small object has bled into the background during denoising, the median of a corrupted region is still wrong, while coherent noise prevents the bleeding. If filtering matches us, the contribution reduces to the calibration analysis.

Why not just feed the image at higher resolution, or tile it? This attacks resolution loss directly. MSR-RCNN and the importance pooling in LIP suggest that more pixels alone do not stop small features from being diluted, but those results are for detection, not depth. We compare against 2× upscaled inference with an equal-compute version of our method. Higher resolution also does nothing for the ambiguity between scale and distance, which is a property of the projection, not of pixel count. So we expect tiling to help with fragmentation and not with calibration.

Why not just reuse the same noise everywhere? ISCS shows that identical noise causes copying artifacts. Our $\rho=1$ endpoint is the in-image analogue, and the sweep will show whether the optimum lies at an intermediate value.

Proposed Experiments

The main experiments run on the KITTI official split and NYU-Depth-V2, using DiffusionDepth's training data and metrics: RMSE, AbsRel, and δ<1.25. Small-object masks come from an off-the-shelf instance segmenter applied to the evaluation images. Error is binned by object area, below 64, 64–256, 256–1024, and above 1024 pixels. Only pixels with valid lidar count, given KITTI's sparsity. For uncertainty we report three measures: per-object Spearman correlation between spread across samples and absolute error, best-of-ten error per object, and within-sample depth variance inside each mask. As a stress test of the resolution argument, we run the same protocol on RS3DBench at its coarser ground resolutions, where buildings and vehicles shrink to a few pixels. There we compare against the Marigold-RS figures from that paper (MAE 23.4, RMSE 29.7 at 30 m resolution). Every method uses three seeds, and we report the variation across seeds.

The decisive check is a clean interaction between object size and method. Content-anchored correlated noise should beat both independent noise and content-blind correlated noise in the smallest bins. The gap should shrink in the largest bins, and full-image RMSE should not degrade. If content-blind correlated noise matches content-anchored noise, the effect is generic low-frequency noise and the object-anchoring claim fails. If average pooling matches importance pooling, the resolution argument fails. If segment-median filtering matches on both accuracy and calibration, the simpler method wins. Each outcome is reportable, and each falsifies one specific part of the claim. Training-free results come first, because a gain without any fine-tuning would show that the effect lives in the noise itself and not in the extra training.