Scale-Free Boundary Graphs for Sim-to-Real Monocular Depth

A single photograph cannot tell you how far away things are in absolute terms. A toy car up close and a real car far away can produce the same image. Yet a photograph says a great deal about relative depth where one object meets another: which one is in front, and by roughly what ratio. We propose to train depth networks entirely in simulation so that they predict only this relative, boundary-located information. The prediction takes the form of a small graph: nodes are object instances, and edges carry the depth jump across the boundary two instances share. The absolute scale that simulation gets wrong, and that a single image cannot pin down, is then left to a single number.

The hook is a simple property of wavelets, which are filters that split a signal into a coarse average plus layers of sharp local detail. Take a wavelet transform of log-depth. Every detail filter sums to zero, so multiplying depth by an unknown scale adds a constant to log-depth and leaves all detail coefficients exactly unchanged. The scale ambiguity of monocular depth therefore lives entirely in the coarsest average band. The detail bands are scale-free. Single Image Depth Prediction With Wavelet Decomposition showed that these detail bands are also sparse and concentrated at object boundaries. If this reasoning holds, the part of depth that simulation can teach reliably is a sparse set of boundary jumps, and a scene graph is the natural container for them.

Core Claim

Our claim is testable. When a model is trained on synthetic data and tested on real images, most of its depth error comes from two sources: the global and coarse-layout band, and texture-driven detail that has no geometric counterpart. The scale-free log-depth jumps at instance boundaries transfer with much less degradation. A model built on this has four parts. It uses wavelet (lossless) downsampling in the encoder. It predicts sparse log-depth detail coefficients. It pools those coefficients along shared instance boundaries into graph edges. It then solves a small least-squares problem on the graph that forces the jumps to be mutually consistent. Such a model should transfer from Virtual KITTI 2 to KITTI, and from Replica to NYUv2, better than dense depth regressors trained on the same synthetic data. It should also do better than adversarial feature alignment.

The consistency constraint is what makes the graph more than bookkeeping. Let $\delta_{ij}$ be the predicted log-depth jump from instance $j$ to instance $i$, averaged over their shared boundary. Let $u_i$ be an unknown log-depth offset for instance $i$. Because log-depth is a function over the image, the jumps must sum to roughly zero around any closed loop of adjacent instances. We recover offsets by

$$\min_{u} \sum_{(i,j)\in E} w_{ij}\,\big(u_i - u_j - \delta_{ij}\big)^2,$$

where $E$ is the set of instance pairs sharing a boundary and $w_{ij}$ is a learned confidence for each edge. This is a graph Laplacian solve. It determines $u$ up to one global constant, which is exactly the monocular scale ambiguity, and it lets errors on individual edges be corrected by the rest of the graph.

How This Differs From Prior Work

Four papers supply pieces of this design, but none combines them.

  • The wavelet decoder. WaveletMonodepth predicts a coarse depth map plus sparse wavelet details and reconstructs with the inverse transform. We reuse that decoder, with two differences. It regresses depth per pixel within a single real benchmark. It never isolates scale in the coarse band, never organizes detail by instance, and never tests sim-to-real transfer.
  • Sim-to-real scene graphs. Sim2SG motivates the split we exploit. It shows the gap has an appearance part and a content part, and it handles appearance with an adversarial layer that forces domain-indistinguishable features. That alignment ignores frequency and works on 2D box relations. We instead pick a representation, scale-free boundary jumps, that we argue is less domain-sensitive by construction, and we measure this directly.
  • Scale and camera normalization. MAGneT-3D constrains how we handle camera differences. It shows that learned queries overfit the training camera's depth distribution, and it fixes this by normalizing depth by focal length. Our coarse-band isolation is a more general version of the same move. Any camera or scale mismatch that shows up as a multiplicative factor is pushed into one scalar.
  • Wavelets for sim-to-real. Light of Normals shows that a wavelet branch for surface detail supports zero-shot transfer from the synthetic PS-Verse dataset to real benchmarks. It works with multi-light captures of isolated objects rather than monocular scenes.

The closest graph-wavelet work is Kumaraswamy Wavelet for Heterophilic Scene Graph Generation. It applies band-pass wavelet filters over graph adjacency to keep fine predicates from being blurred together. Our edges come from spatial wavelets of depth, not graph spectra. We borrow its warning that low-pass message passing blurs sharp relational differences, which is why we use an explicit least-squares solve rather than many rounds of neighbor averaging.

Two papers argue against scale alone being enough, and we take both seriously. Towards Ambiguity-Free Spatial Foundation Model shows that large depth models commit to a single surface at glass and similar multi-layer regions. It recovers the hidden layer with a high-frequency Laplacian prompt, which is further evidence that high-frequency structure carries depth information that ordinary predictions suppress. MonoScene avoids committing to depth by spreading features along camera rays into voxels. By its own account, that spreading hurts small and thin structures, which are exactly the boundary regions we target.

Research Program

The program has three stages.

Stage 1: measurement. Before building anything, we train a standard encoder-decoder depth network and a WaveletMonodepth-style network on synthetic data only, using Virtual KITTI 2 and Replica. We evaluate both on the paired real sets. We then break the log-depth error into wavelet bands and into boundary versus interior pixels. We also compute image-side wavelet detail energy in synthetic versus real images at the same locations.

The claim predicts that real-domain error concentrates in the coarse band and in detail coefficients away from true depth edges. If instead the boundary detail coefficients degrade as much as the coarse band, the core premise is false, and the project stops or turns to multi-layer boundaries.

The choice of Virtual KITTI 2 is deliberate. It re-rendered its predecessor with physically based materials and lighting and roughly halved the sim-to-real gap. That leaves a remaining gap that is modest, so it can be analyzed.

Stage 2: the model. The encoder replaces strided downsampling with Haar wavelet downsampling. This follows the argument in Framing U-Net via Deep Convolutional Framelets that ordinary pooling discards the detail needed for exact reconstruction.

The decoder predicts two things: a coarse log-depth map and sparse detail coefficients of log-depth. The detail coefficients are supervised directly against the wavelet transform of synthetic ground truth. Scale-invariant, so no synthetic scale bias enters them.

  • In simulation, instance masks come from ground truth.
  • At real test time, they come from an off-the-shelf segmenter.

We pool detail coefficients along each shared boundary into $\delta_{ij}$, with a small network predicting $w_{ij}$ from the pooled coefficients and local image features. We then solve the least-squares problem above for per-instance offsets. The final depth combines four components:

  • the per-instance offsets;
  • within-instance smooth shape from the coarse band, with its mean removed;
  • boundary detail;
  • one global scale, set from camera height or reported with scale-invariant metrics.

Training back-propagates through the linear solve, which is cheap because instance graphs are small.

Stage 3: real test time. We ask whether the graph solve can be adapted to real images without labels. Enforcing cycle consistency on unlabeled real images gives a self-supervised signal: the loop residual. This fits naturally with the noise-aware pseudo-labeling ideas of CTS, which showed that sim-to-real pseudo-labels must be filtered by uncertainty before they help.

Baselines

Do-nothing reference. A conventional depth network with strided downsampling and per-pixel scale-invariant log loss, trained on synthetic data only and evaluated on real data. We evaluate it both with ordinary median-scaling at test time and without. Every other number is read against this gap.

Strongest method in our training regime: sim-only WaveletMonodepth. WaveletMonodepth already predicts sparse wavelet detail and is the closest architecture. The file reports that, on KITTI, discarding 90% of its detail coefficients costs under 1.4% relative accuracy, and that on NYUv2 discarding 95% costs 0.2%. We found no reported result on Lacuna for WaveletMonodepth, or any other method here, trained on Virtual KITTI 2 or Replica and tested on KITTI or NYUv2. We will run it ourselves with the same data, the same backbone depth and parameter count, and the same hyperparameter search budget as our model. Beating it means lower real-domain scale-invariant error and higher pairwise instance ordering accuracy, with seed variance reported.

Adversarial alignment. A gradient-reversal domain classifier on the encoder, which is the appearance-gap tool from Sim2SG. We will give it the unlabeled real images our Stage 3 variant sees, so that the two are compared under equal access to real data.

Foundation-model reference. Depth Anything v2, evaluated zero-shot and with its predicted depth converted into our boundary jumps. This is not a fair competitor, because it was trained on large amounts of real imagery. We report it as a ceiling and as a test of whether our edge head adds value on top of strong features. For this model, the file's only reported number is 75.5% multi-layer spatial relationship accuracy on MD-3k when combined with Laplacian prompting (Towards Ambiguity-Free Spatial Foundation Model). We found no reported result on Lacuna for it on KITTI or NYUv2 depth metrics.

Ablations. These isolate the key ingredient:

  • The same model with the graph solve removed, so boundary detail is used per pixel.
  • Edges built from image detail coefficients instead of predicted depth detail.
  • Supervision on raw depth detail rather than log-depth detail, which breaks exact scale invariance.
  • Strided convolution in place of wavelet downsampling.

If removing the graph solve costs nothing, the structured part of the proposal is unnecessary.

Why not just…?

Why not just use heavy augmentation or domain randomization? It is cheap and it is the default. The file's evidence is that it falls well short on structured outputs. On KITTI's hard setting, spatial domain randomization in Sim2SG reaches 0.234 mAP against 0.316 for structured transfer. Photometric augmentation also cannot address the scale mismatch, which is geometric. We will still apply strong augmentation to every model, including ours, so any gain we report is on top of it.

Why not just take a strong foundation depth model and read off relative depth between instances? This is a real threat for anyone allowed to use real training data. Our question is narrower: what can be learned from synthetic supervision alone, which matters for sensors, environments, or labels that exist only in simulation. We will test it directly. If Depth Anything's features plus our edge head, fine-tuned only on synthetic data, beat Depth Anything alone on real ordering accuracy, the representation earns its place. If not, we will say that the contribution is limited to training from scratch in simulation.

Why not just low-pass the input images to strip simulated texture? This is the naive version of the frequency idea, and it is plausible. It also removes the image edges that signal depth discontinuities, so we expect it to damage boundary accuracy. Stage 1 tests this directly by comparing image detail against depth detail at the same pixels, and we include a low-passed-input baseline.

Why not just train with a scale-invariant loss, which already ignores global scale? A scale-invariant loss removes a single global factor. It still asks the network to regress coarse layout from synthetic statistics, and it gives no mechanism to reconcile inconsistent local predictions. The ablation that keeps log-depth detail supervision but drops the graph solve separates the two effects. If it matches the full model, the graph adds nothing and we will report that.

Proposed Experiments and Decisive Checks

The main transfer pairs are Virtual KITTI 2 to KITTI (outdoor, sparse LiDAR ground truth) and Replica to NYUv2 (indoor, dense ground truth for boundary evaluation). A ScanNet test set checks whether results generalize across real indoor sensors.

We use four kinds of metrics:

  • standard scale-invariant log error and threshold accuracy;
  • pairwise ordinal accuracy between adjacent instances, using LiDAR or depth-sensor points inside masks;
  • boundary precision and recall of depth discontinuities on NYUv2;
  • cycle residual as a diagnostic that needs no labels.

Four results decide the outcome:

  • Stage 1 decomposition. It must show that boundary detail bands degrade from simulation to reality substantially less than the coarse band. Otherwise the premise fails.
  • Graph ablation. The full model must beat both sim-only WaveletMonodepth and itself-without-graph on real ordinal accuracy and boundary recall, across at least three seeds.
  • Segmenter dependence. We swap ground-truth masks for predicted masks at real test time. The directions summary warns that structured pipelines fail when upstream segmenters miss objects, so we report how gains shrink with segmenter quality.
  • Multi-layer extension. On MD-3k we check whether allowing two edges per boundary recovers glass layers without the input-side Laplacian prompt. This is framed as an extension, not a headline claim, since the file reports only dual-layer results for prompting.

A negative outcome on the first check is itself useful. It would show that sim-to-real depth error is not confined to scale and texture, contrary to what both the focal-normalization and the wavelet-transfer literatures implicitly assume.