Albedo-Swapped Synthetic Training for Guided Depth Super-Resolution

Cheap depth sensors, such as the time-of-flight cameras in phones, return coarse and noisy depth maps. Guided depth super-resolution sharpens them using a high-resolution color photo of the same scene. The color photo helps because depth edges usually coincide with color edges. The trouble is that many color edges are not depth edges at all: printed text, wallpaper, decals and shadows lie on flat surfaces. Networks trained on real data learn the shortcut "color edge implies depth edge" and stamp these patterns into the geometry, an error the field calls texture copying. This is a symptom of an ill-posed inverse problem, one where many high-resolution depth maps are consistent with the same low-resolution input. When the low-resolution measurement cannot settle which answer is correct, the network falls back on whatever correlation its training data offered.

We propose to train these networks only on rendered scenes. In each training pair, the same 3D geometry is rendered twice, under the same lighting, with two unrelated random surface textures. The network must output the same depth for both renders. Under this pairing, a color edge carries no information about depth unless it comes from shading or from an occlusion boundary. The low-resolution depth input is degraded by a randomized simulator of real sensor faults. Some of those faults, such as missing returns on dark surfaces, depend on the randomized texture, so the network also sees realistic correlations between appearance and sensor failure. The question is whether this purely synthetic, texture-decoupled training can approach the best networks trained on real paired captures in overall error while copying clearly less texture.

Core Claim

Texture copying is not mainly an architecture problem. It comes from a spurious correlation in the training distribution, and it can be removed by changing that distribution so that texture and geometry vary independently. The proposal combines three directions, each in a distinct role. Synthetic-to-real transfer supplies the mechanism: randomization over appearance, following Domain Randomization for Scene-Specific Car Detection and Pose Estimation, which used random textures to force car detectors onto shape. It also draws on the finding in Diversity is Definitely Needed that diversity of synthetic data matters more than realism. Cross-modal fusion is the application setting, with RGB-guided depth. The ill-posed nature of super-resolution explains why the shortcut appears and where it does the most harm.

The training objective for a network $f$ is

$$\mathcal{L} = \|f(y_a, I_a) - x\|_1 + \|f(y_b, I_b) - x\|_1 + \lambda\,\|f(y_a, I_a) - f(y_b, I_b)\|_1 .$$

Here $x$ is the rendered high-resolution depth. $I_a$ and $I_b$ are color renders of identical geometry and lighting with two independently sampled textures. $y_a$ and $y_b$ are the corresponding simulated low-resolution sensor depths, which can differ because dropouts depend on texture. $\lambda$ weights the cross-texture agreement term. The third term isolates the new ingredient. Plain domain randomization supplies only the first two terms, and the ablations remove the third term or the pairing.

Research Program

We first build a rendering engine for indoor scenes. Procedurally arranged CAD furniture and room shells are rendered with physically based shading. Each geometry and light configuration gets two or more texture draws. Textures come from a large random bank that includes text, high-contrast stripes, checkerboards and natural photos, in the spirit of the random texture banks of Khirodkar et al.. The research file lists no ready-made synthetic RGB-D super-resolution set, so this engine is a contribution in itself.

Shading is kept physically consistent rather than replaced by flat texture. Photometric Depth Super-Resolution shows that shading in the color image carries real high-frequency surface information. That paper had to assume either piecewise-constant reflectance or a face-specific reflectance network to separate shading from texture. We want the network to learn that separation from data instead.

Second, we build the sensor degradation simulator. It follows the high-order randomized pipeline of Real-ESRGAN, which trains on purely synthetic degradations yet restores real images. We extend it with the range-sensor faults that DORNet and EGP3D identify as missing from synthetic pipelines:

  • dropouts at low reflectivity, driven by the rendered texture;
  • flying pixels and multipath blending at depth discontinuities;
  • stray-light offsets;
  • non-uniform blur;
  • small random errors in the alignment between the depth sensor and the color camera.

The parameters of each fault are sampled over wide ranges rather than fit to one sensor.

Third, we train existing guided depth networks under this regime without changing their architectures. These are DORNet, the structure-flow network SFG, and the implicit-function network JIIF. Keeping the architectures fixed tests the claim that the data, not the network, causes texture copying.

Baselines

The naive reference point is upsampling without any guide, bicubic interpolation of the low-resolution depth. The second reference is each architecture trained the standard way on bicubically downsampled NYU-v2. This corresponds to JIIF's setting, where it reports RMSE 5.27 at ×16 on NYU-v2 (JIIF). This baseline tells us how far a clean-downsampling model falls when moved to a real sensor.

The strongest existing competitor is DORNet trained on real paired RGB-D-D captures. It learns the sensor's degradation without labels and uses that estimate to decide where color guidance is injected. It reports RMSE 3.42 cm on RGB-D-D, ahead of SFG at 3.88, DKN at 5.08 and FDSR at 5.34 (DORNet; SFG). We rerun DORNet with its released training recipe. Our variants use the same architecture, the same number of optimizer steps and the same hyperparameter search budget, with the RGB-D-D training pairs replaced by synthetic pairs.

We count the following as beating it. With zero real training pairs, the synthetic model should come within roughly 10% of DORNet's RGB-D-D RMSE and show a clearly lower phantom-edge rate, a measure defined below. With brief fine-tuning on real RGB-D-D, it should improve on 3.42 cm in RMSE.

Two ablations isolate the key ingredient. The first removes the cross-texture agreement term but keeps random textures, which amounts to plain domain randomization. The second keeps the agreement term but replaces physically re-rendered textures with textures pasted in 2D onto the rendered color image after shading. A third ablation swaps our sensor simulator for Real-ESRGAN-style generic blur, noise and downsampling.

Why not just…?

Why not just train on real paired captures, as DORNet does? Real pairs encode the true sensor, and DORNet's 3.42 cm shows they work. However, real indoor scenes also contain the texture–geometry correlation that causes copying, and paired captures with a high-end reference sensor are expensive to collect for each new device. Our cross-sensor tests address this directly. They check whether a DORNet trained on the Huawei sensor in RGB-D-D degrades on TOFDSR/ToFMark, and whether the synthetic model degrades less there. No reported result was found on Lacuna for DORNet on TOFDSR. This alternative remains a real threat on in-distribution RMSE, which is why fine-tuning on real data is part of the comparison.

Why not just paste random textures onto real or rendered color images in 2D? This is the most serious threat. It is far cheaper than rendering and also breaks the color-edge-to-depth-edge shortcut. Its weakness is that pasted texture ignores shading, foreshortening and texture-dependent dropouts. It therefore teaches the network to distrust all fine color structure, including the shading that carries real surface detail. The second ablation tests this directly. If 2D pasting matches re-rendering in both RMSE and phantom-edge rate, we will report the cheaper recipe as the finding and drop the claim that physical rendering is needed.

Why not just use the deep-feature guidance of DADA, which was designed to keep texture edges from blocking smoothing? DADA predicts from deep color features how freely depth may be smoothed between neighboring pixels. It then enforces exact agreement with a known downsampling operator, and reports MSE 223 at ×32 on NYU-v2 versus 253 for LGR. Its features are still learned from bicubic NYU-v2 data, where the shortcut exists. Its consistency step also assumes the wrong degradation operator on real sensors. We include DADA as an additional baseline under both training regimes to see whether its design alone suppresses copying.

Why not just add an uncertainty head so the network can discount color guidance where it is unreliable? Heteroscedastic uncertainty, as in Tanno et al., lets a network down-weight regions it cannot predict well. A network that confidently copies texture, however, is not uncertain there. Uncertainty addresses noisy regions, not wrong correlations. We will add such a head to the strongest model and test whether its uncertainty is high on phantom edges. We expect it not to be.

Proposed Experiments

Main evaluation. We use real RGB-D-D at its standard protocol and report RMSE in cm. We also report TOFDSR/ToFMark; no numbers were found on Lacuna for those, so we rerun all baselines there. For the clean-downsampling regime, we report NYU-v2 at ×16 and Middlebury at ×8. On Middlebury, the guided-depth survey quotes RMSE 1.56 for AHMF and 5.41 for guided filtering (Guided Depth Map Super-Resolution: A Survey).

Texture-copying metric. RMSE barely registers texture copying, so we define a phantom-edge rate. It is computed over pixels where the color gradient is strong but the ground-truth depth gradient is near zero. For each method we report the mean predicted depth-gradient magnitude in those pixels, and the fraction of them where predicted depth deviates locally from a fitted plane.

Controlled probe. We also build a probe set from real RGB-D-D frames. On known-planar regions we print or project high-contrast patterns, so the true geometry is fixed while appearance changes. Prediction differences across patterns then measure copying directly.

Decisive checks. The proposal succeeds if three things hold. The paired agreement term must reduce the phantom-edge rate beyond plain randomization, which tests whether pairing matters. Physically re-rendered textures must beat 2D pasting on fine-structure surfaces, which tests whether rendering matters. And zero-real-pair training must stay close to DORNet's 3.42 cm while copying less, which tests whether the approach is practical.

Failure condition. The proposal fails if both of the following hold. Synthetic training leaves RMSE far behind real-pair training even after fine-tuning, and the phantom-edge gains vanish once a real-pair DORNet is evaluated on the probe set. Either outcome is informative about whether texture copying lives in the data or in the architecture.