Where the Depth Could Flip: Oblique-Tree Heads for Sim-to-Real Depth Reliability

A depth network trained in a simulator and then run on real camera images fails unevenly. Most pixels transfer well, but a minority break badly. These are often the pixels where a single image cannot settle depth on its own: a small near surface looks exactly like a large far one, and the network had to guess. This proposal replaces the final prediction layer of a monocular depth network with an oblique decision tree, meaning a tree whose splits test weighted sums of features ($\mathbf{w}^T\mathbf{z} > c$) rather than one feature at a time. Such a tree cuts feature space into convex polyhedral cells, each with its own linear depth predictor. Because every cell boundary is an explicit hyperplane, we can compute exactly, without any sampling, how far a pixel's feature vector is from a boundary. We can also compute how much the predicted depth would jump if the feature crossed it.

The central bet is that, measured against a realistic budget for how far real-sensor appearance changes move features, these distances predict where sim-trained depth will fail on real data. They should do so more cheaply than sampling-based uncertainty and at least as reliably. We test this against real ground truth on a physical phantom rather than only inside a simulator. In two sentences: make the last layer of a depth network a tree of hyperplanes, so that "how close is this pixel to switching to a very different depth" becomes an exact geometric quantity. Then check, on real measured depth, whether that quantity flags sim-to-real failures better than dropout, ensembles, or a learned variance head.

Core Claim

Given a frozen depth backbone trained on synthetic data, we fit an oblique regression tree with linear leaves on reduced per-pixel features. For a pixel feature $\mathbf{z}$ in leaf $L$, the exact distance to leaving that leaf is the smallest hyperplane distance along its root-to-leaf path:

$$\rho(\mathbf{z}) = \min_{(\mathbf{w},c)\in \text{path}(L)} \frac{|\mathbf{w}^T\mathbf{z}-c|}{\|\mathbf{w}\|}.$$

For a displacement radius $r$, the depth interval is $[\underline{D}_r(\mathbf{z}), \overline{D}_r(\mathbf{z})]$, taken as the minimum and maximum of each leaf's linear prediction over that leaf's polytope intersected with the ball of radius $r$ around $\mathbf{z}$. This is a small convex program per reachable leaf, of the same kind Counterfactual Explanations for Oblique Decision Trees: Exact, Efficient Algorithms solves in 40–120 ms for classification counterfactuals. The new ingredient is the choice of $r$. Rather than assuming isotropic noise, we estimate the distribution of feature displacements from paired images that share the same geometry but differ in appearance. We obtain these pairs by rendering simulator frames and translating them toward real texture with shading-anchored translation. The claim is that the interval width at $r$ taken as a high quantile of that displacement distribution ranks real-data depth errors better than competing uncertainty signals, at similar compute.

Depth ambiguity appears concretely in this picture: an ambiguous pixel is one whose feature sits near a hyperplane separating leaves that predict very different depths. Sensor noise appears as the radius $r$. Real-world validation is the test.

Relation to Prior Work

The method comes from the sparse oblique tree line of work. Sparse oblique decision trees: a tool to understand and manipulate neural net features replaces a CNN classification head with a tree trained by alternating optimization (TAO). It reaches 7.90% error against VGG16's 6.79% on a 16-class ImageNet subset while using 17% of the features. Faster Neural Net Inference via Forests of Sparse Oblique Decision Trees compresses whole layers with oblique forests. Both works handle only discrete classification on clean benchmarks, and neither uses the tree geometry for reliability under shift. We move to dense continuous regression and use the geometry as the product, not the compression. Linear-leaf oblique trees follow Exact Learning of Linear Model Predictive Control Laws using Oblique Decision Trees with Linear Predictions, which fits piecewise affine laws on low-dimensional states. As a variant with continuous predictions, we use Hinge Regression Tree: A Newton Method for Oblique Regression Tree Splitting. On tabular regression it matches CART's error with 5.8 leaves against 113, and its cubic cost in dimension is manageable because we reduce features to about 32 dimensions.

Two results motivate the proposal. Uncertainty Quantification with Statistical Guarantees in End-to-End Autonomous Driving Control found that Monte Carlo dropout's safety score collapsed to 0.00 under simulated rain. Hamiltonian Monte Carlo stayed calibrated but is too slow for real-time use, and all of this was shown only in CARLA. That leaves room for a signal that is both exact and cheap. Adaptive Monocular Depth Estimation with Masked Image Consistency reports that augmentation-consistency adaptation often does worse than the unadapted source model on depth. This constrains us against relying on generic image augmentation to model the shift.

The application setting and the source of structured noise come from PPS-Ctrl: Controllable Sim-to-Real Translation for Colonoscopy Depth Estimation. It trains depth on synthetic SimCol3D, translates frames with a shading-conditioned diffusion model, and evaluates against real phantom ground truth on C3VD. PPS-Ctrl improves the depth predictor itself. It says nothing about which pixels to trust, and its translations are what we reuse to measure $r$. The simulation-only concern is motivated by Deep Local Trajectory Replanning and Control for Robot Navigation, which was validated only in Gazebo. The case for a geometric bottleneck comes from Deep Drone Acrobatics (Extended Abstract), where abstraction-based policies reached 100% success against 0% for raw-image policies. We borrow that intuition but place the bottleneck at the prediction head.

Research Program

We freeze a DepthAnything backbone trained on SimCol3D, following PPS-Ctrl's recipe, and add a 1×1 projection to $k=32$ channels trained jointly with a standard linear head. We then fit the tree head on subsampled pixel features, with depth targets taken from the backbone head or from ground truth. There are two variants: TAO-trained sparse oblique trees with linear leaves, and HRT. Depth is limited to about 8, giving at most about 256 leaves.

To build the displacement budget, we pass each SimCol3D frame and its PPS-Ctrl translation through the backbone. We record per-pixel feature displacement, projected onto each pixel's local split normals, and take quantiles per image region and per depth bin.

At test time, $\rho$ is closed form and costs little more than the forward pass. The interval program runs only over leaves reachable within $r$, using the path hyperplanes for pruning. We will report wall-clock cost per frame.

We also study the evaluation question directly. We rank all uncertainty methods twice: on SimCol3D with synthetic corruptions, and on real C3VD. We then measure whether the two rankings agree.

Baselines

The naive reference is the unmodified DepthAnything backbone and head trained on SimCol3D and PPS-Ctrl translations, with no uncertainty signal. Its "uncertainty" is predicted depth itself, which is a reasonable proxy given that error grows with distance. Any method that cannot beat depth-as-uncertainty at ranking errors is worthless.

The strongest existing depth pipeline is PPS-Ctrl. It reports a 20% RMSE improvement for DepthAnything trained on its translations over GAN-based translations on C3VD, and a structural error of 0.092 against 0.156 for depth-based control. The file gives no absolute C3VD RMSE, so we will reproduce the absolute number ourselves. The accuracy criterion is that our tree head's RMSE stays within 5% relative of the reproduced PPS-Ctrl model.

For uncertainty on C3VD we found no reported result on Lacuna. We therefore implement four competitors on the same backbone, the same training data (including translated pairs), and the same tuning budget of 20 configurations each:

  1. Monte Carlo dropout on the head.
  2. A five-member deep ensemble of heads.
  3. A heteroscedastic head that predicts a variance and is trained with Gaussian likelihood on both synthetic and translated frames. This is the most dangerous competitor.
  4. Test-time augmentation.

Beating them means lower area under the sparsification error curve (AUSE) and higher AUROC for detecting pixels with absolute relative error above 0.2 on C3VD, at equal or lower per-frame compute.

Three ablations isolate the key ingredients. The first computes the same interval from a first-order linearization of an MLP head ($|\nabla D|\cdot r$), which tests whether exact polyhedral geometry matters or any gradient-based sensitivity would do. The second replaces the translation-derived $r$ with an isotropic Gaussian-noise radius, which tests whether structured displacement matters. The third uses an axis-aligned tree of equal leaf count, which tests whether oblique splits matter.

Why not just…?

Why not just use a deep ensemble or a variance-predicting head? This is the real threat. A heteroscedastic head trained on translated frames may capture exactly the pixels we flag. The evidence against assuming it will is limited: Uncertainty Quantification with Statistical Guarantees in End-to-End Autonomous Driving Control shows cheap approximations can collapse under sensor shift, but ensembles and variance heads were not tested there. We therefore give these baselines the same paired data and let C3VD decide. If the variance head matches us, the tree's remaining value is exactness and auditability, and we will report that as the finding rather than claim superiority.

Why not just a gradient-norm sensitivity on a normal MLP head? Linearization assumes the prediction changes smoothly. The failure we target is a discrete jump to a different depth mode, which a local gradient at $\mathbf{z}$ cannot see. The first-order ablation measures this directly: if it performs as well as the tree, oblique geometry adds nothing and the claim fails.

Why not just use axis-aligned trees or boosted trees on the features? Axis-aligned cells approximate correlated feature boundaries with staircases. Hinge Regression Tree: A Newton Method for Oblique Regression Tree Splitting needed 113 CART leaves against 5.8 hinge leaves on tabular data. That fragmentation creates many spurious nearby boundaries and inflates intervals everywhere. The equal-leaf axis-aligned ablation tests whether this hurts error ranking in practice.

Why not just improve the translation so there is nothing to flag? Better translation helps accuracy, but PPS-Ctrl still struggles with specular highlights, fluid, and small structures. Uncertainty is needed precisely where translation fails. We report both accuracy and error ranking, so any gain from translation alone is visible.

Proposed Experiments

The main experiment trains on SimCol3D and PPS-Ctrl translations and tests on C3VD real phantom depth. It reports RMSE and AbsRel for accuracy; AUSE, AUROC at the 0.2 AbsRel threshold, and coverage and width of the $[\underline{D}_r,\overline{D}_r]$ interval against real depth; and latency on a single GPU. Colon10K has no ground truth and is used only for qualitative inspection of flagged regions.

A secondary transfer study renders CARLA frames with depth, trains on them, and tests on the KITTI Eigen split and DDAD. This checks that the effect is not specific to colonoscopy, including the far-range regime where depth error grows fastest.

The decisive checks are three. First, the tree head must keep RMSE within 5% of the reproduced PPS-Ctrl model; otherwise the reliability signal describes a worse predictor and does not count. Second, tree intervals must beat the variance head and the ensemble on C3VD AUSE, with 95% bootstrap confidence intervals over sequences that do not overlap; a tie refutes the superiority claim. Third, the translation-derived $r$ must beat the Gaussian $r$ and the first-order ablation must lose; otherwise the two components that make the idea new are not doing the work.

Finally, the ranking study reports Kendall's τ between method rankings on corrupted SimCol3D and on C3VD. A low τ would show directly that simulator-only evaluation would have picked the wrong uncertainty method, and would quantify the concern raised by Gazebo-only and CARLA-only studies.