A camera-only 3D car detector trained in a driving simulator usually falls apart on real roads. The common explanation is that rendered images do not look real. This proposal tests a narrower hypothesis. For a single camera, the depth cue that should transfer from simulation to reality is geometry measured in pixels: how tall a car appears given how tall cars are, and where its wheels touch the road given how high the camera sits. Both are pure projective geometry, and a simulator reproduces them exactly. Such cues fail to transfer for two reasons. Networks trained on synthetic data read depth from rendering shortcuts instead. And at long range the pixel measurements are only a few pixels tall, so the network's internal downsampling blurs them away, while real optics blur them differently from the renderer.
We propose a detector whose only route to depth runs through a small set of 2D pixel measurements followed by a closed-form geometric lift. A dedicated full-resolution pathway handles distant objects. The same geometry lets us compute a label-free confidence for each detection, which decides which automatically generated labels are trustworthy on real images. The pitch to a colleague is short. Force the network to measure depth the way a surveyor would, in pixels and known sizes, because simulators get pixels and sizes right. Then spend the network's resolution budget where those pixel measurements are tiny, which is exactly where the real-world gap lives.
Core Claim
We claim three things. First, the sim-to-real gap for monocular 3D detection is concentrated in the far field and is dominated by error in 2D measurements, not by error in the geometric lift. Second, routing all depth through a geometric bottleneck makes the part of the model trained on synthetic data transfer far better than a free-form depth head does at matched accuracy in simulation. Third, the known geometry supplies an uncertainty for each object that grows with the square of distance, and this beats a learned uncertainty for filtering pseudo-labels on unlabeled real images.
The bottleneck works as follows. The network predicts, for each detected object, a pixel height $h$, the image row of its ground contact point $v_b$, a physical height $H$ drawn from a class-conditional size prior, and a yaw. Depth comes from two independent estimates, $Z_{\text{size}} = fH/h$ and $Z_{\text{ground}} = f\,h_c/(v_b - v_0)$. Here $f$ is the focal length in pixels, $h_c$ the camera mounting height, and $v_0$ the horizon row implied by camera pitch. The two estimates fail in different ways: the size cue fails when the size prior is wrong, and the ground cue fails when the road is not flat. Propagating a pixel error $\sigma_h$ and a size-prior spread $\sigma_H$ through the size cue gives
$$\frac{\sigma_{Z}}{Z} \approx \sqrt{\left(\frac{\sigma_h}{h}\right)^2 + \left(\frac{\sigma_H}{H}\right)^2}, \qquad \sigma_Z \approx \frac{Z^2 \sigma_h}{fH} \text{ when pixel error dominates,}$$
which follows from first-order error propagation of $Z = fH/h$. The analogous expression for the ground cue follows the same way. For pseudo-labeling on real images, we keep a detection only if the two depth estimates agree within their combined analytic uncertainty:
$$|Z_{\text{size}} - Z_{\text{ground}}| < \kappa \sqrt{\sigma_{Z,\text{size}}^2 + \sigma_{Z,\text{ground}}^2},$$
with $\kappa$ a single threshold tuned on synthetic validation data. On real data, $\sigma_h$ is estimated without labels. We shift the input by fractions of a pixel, re-run the model, and use the spread of the resulting $h$ values.
Relation to Prior Work
Work on depth-based sim-to-real transfer motivates the premise that geometry is the transferable abstraction. Adapting control policies from simulation to reality using a pairwise loss replaces RGB with depth to reduce lighting sensitivity. Pixel-Attentive Policy Gradient for Multi-Fingered Grasping in Cluttered Scenes transfers depth-only grasping policies with little loss. Habitat reports that depth keeps navigation agents from overfitting to textures. All three assume a depth sensor at test time. We keep their premise but relocate it: the invariant geometry is not a depth map from a sensor but the pixel measurements that projective geometry turns into depth.
The closest work on sim-to-real 3D detection constrains our design. CTS adapts LiDAR detectors from CARLA to KITTI using a teacher network that generates labels for a student, filtered by a learned uncertainty over box corners. That method relies on directly measured range. In the monocular case, a learned uncertainty trained on synthetic data has no guarantee of staying calibrated on real images, whereas our analytic uncertainty is tied to geometry that does not change across domains. Transfer Learning from Simulated to Real Scenes for Monocular 3D Object Detection pre-trains on 1.4 million synthetic RoadSense3D images but needs labeled real fine-tuning. We target the unlabeled setting. Aug3D-RPN synthesizes new object distances by warping real images with real LiDAR, so it is not simulator transfer. DA-3Ddet aligns monocular features to real LiDAR features, which again requires privileged real range data.
Monocular geometry methods supply parts of our method, but all are trained and tested on real labels. CaDDN predicts a probability distribution over depth bins that widen with distance. That choice deliberately coarsens far-field depth, the regime we care most about. MonoHBD shows that pixel-wise depth losses are dominated by large nearby objects and fixes this with fixed-size features cropped around each object. We reuse that insight: our losses are defined per object, not per pixel. AutoShape and DCD recover depth from many 2D-3D keypoint correspondences. Their gains plateau around 16 and 73 keypoints respectively. Both need sub-pixel keypoint accuracy that downsampling destroys on distant cars, and we deliberately use the two most robust geometric measurements instead of dozens of fragile ones. Generalizing Monocular 3D Object Detection comes closest in spirit. Its CHARM3R component combines regressed depth with ground-plane depth so that errors cancel under changes in camera height, and it uses CARLA as a stress test. It does not train on simulation for real deployment, does not isolate far-field resolution, and does not use cue agreement to filter labels.
On resolution, LIP replaces uniform pooling with learned importance weighting and raises small-object AP on COCO for Faster R-CNN from 21.9% to 24.0%. It is our main architectural competitor for the far-field pathway. Virtual KITTI 2 upgraded rendering and closed about half of the sim-to-real tracking gap. That result motivates us: better rendering is not enough. It also serves our experiments, because its scenes clone real KITTI sequences.
Research Program
The work has three parts, and the diagnostic comes first because it can falsify the premise cheaply.
The first part is the error decomposition. Virtual KITTI 2 re-renders real KITTI drives with the same layout and camera, so the same car can often be seen twice, once rendered and once real. We train a standard monocular detector on the rendered frames. On real frames, we then split its depth error into three parts: error in 2D pixel measurements, error in the size prior, and error from the lift itself. We bin all three by range and by the object's pixel height. The hypothesis predicts that measurement error dominates and grows sharply below roughly 25 pixels of object height. It also predicts that the rendered-versus-real gap in measured $h$ for the same car correlates with a measured difference in edge sharpness between renderer and real camera. If instead the lift or the size prior dominates, the far-field pathway is the wrong investment. We would then redirect the work toward adapting the size prior.
The second part is the model. A shared backbone produces 2D detections. For objects whose predicted pixel height falls below a threshold, a far-field head re-crops the full-resolution input around the object. It then processes the crop at an output stride of 2 to 4 and regresses sub-pixel top and bottom edges and the contact row. Nearby objects use the ordinary downsampled features. Because far objects are few, this adds little compute. Depth is produced only by the two closed-form cues, fused by inverse-variance weighting. There is no free-form depth head that could learn rendering shortcuts. The synthetic training images are degraded with a blur and noise model fitted to unlabeled real KITTI images, using edge-profile statistics at object boundaries. This keeps the pixel measurements statistically matched across domains.
The third part is adaptation without labels. The synthetic-trained model acts as a teacher on unlabeled real KITTI training images. Its pseudo-labels are filtered by the cue-agreement test above and weighted by analytic uncertainty. A student is then trained on them, and teacher and student are updated in the usual moving-average fashion.
Baselines
The naive reference is a monocular detector trained only on synthetic data and applied to real KITTI with no adaptation. We implement it as the same backbone with a direct depth head, using per-object losses in the style of MonoHBD so that the comparison is not handicapped by the near-object bias. Its real-domain score measures the raw gap.
The strongest existing sim-to-real method is CTS-style teacher-student adaptation, ported to the monocular detector: same backbone, same unlabeled real images, learned uncertainty over box corners, and CTS's curriculum sampling. For reference, CTS reports CARLA3D→KITTI $AP_{3D}$ at Moderate difficulty rising from 4.06% to 45.28%. That number is with LiDAR input, so it is an upper reference for what adaptation achieves when range is measured directly, not a monocular target. We found no reported result on Lacuna for monocular 3D detectors trained on CARLA or Virtual KITTI 2 and evaluated on KITTI 3D. Our naive baseline and this ported CTS baseline will establish those numbers.
The strongest monocular architecture comparisons are CaDDN and MonoHBD, each trained on the same synthetic data with the same degradation augmentation and adaptation loop. Only the depth representation differs from our model. MonoHBD reports 0.52 versus 0.40 AP in the 40–60 m band against MonoCon and MonoDETR on real supervised KITTI (MonoHBD). This is the far-field figure our method must approach without any real labels.
The oracle is the same detector trained with real KITTI labels. For context, DCD reports 15.90% Moderate AP on the KITTI test server. Our headline quantity is the fraction of oracle performance each method recovers, measured on the same validation split.
For fairness, every method uses the same backbone, input resolution, synthetic source data, unlabeled real images, and number of training steps. Each gets the same hyperparameter search budget, selected on synthetic validation data only, since real labels are forbidden for tuning. The far-field head is compared at matched inference compute.
Three ablations isolate our ingredients. The first restores a free-form depth head alongside the geometric cues, which tests the bottleneck. The second replaces the far-field head with LIP pooling in the backbone at matched compute, which tests where resolution should be spent. The third swaps cue-agreement filtering for a learned uncertainty filter, which tests the analytic uncertainty. Beating the baselines means a higher real-domain $AP_{3D}$ at KITTI Moderate. It also means the gain is concentrated in the 40–60 m band and holds across three training seeds, with confidence intervals that do not overlap.
Why not just…?
Why not just blur and downsample the synthetic images during training? This is the most serious threat. If our diagnosis is right, degradation augmentation alone could remove much of the gap. We therefore give the augmentation to every baseline, including the naive one. Our claim survives only if the bottleneck and the far-field head add a gain on top of it. If augmented direct-depth models match our model, the honest conclusion is that sharpness mismatch was the whole story, and we would report that.
Why not just increase input resolution or use LIP-style pooling everywhere? Full-resolution feature maps are expensive, as the memory concerns behind 3DZip illustrate. LIP improves small objects but spends effort across the whole image. The matched-compute ablation settles whether targeted re-cropping of far objects outperforms improving the pooling across the whole backbone.
Why not just run CTS-style teacher-student adaptation on a standard monocular detector? CTS filters labels with an uncertainty learned in simulation. With monocular input, depth error grows with the square of range, and a synthetic-trained uncertainty head has no reason to stay calibrated on real images. The filter-swap ablation measures this directly. We report pseudo-label depth error by range under each filter as well as final AP.
Why not just render more realistically? Virtual KITTI 2 is already the photorealistic option, and it closed only about half the tracking gap. We use it as the primary source, so realism is held constant and cannot explain our gains.
Why not just fine-tune on a few real labels? Mohamed et al. show real fine-tuning helps but still needs 3D annotation, which is scarce at long range. As a secondary result, we report our method plus 1% and 5% of KITTI labels against the naive model with the same labels.
Proposed Experiments
Virtual KITTI 2 is the main source, with KITTI 3D as the target. Evaluation uses $AP_{3D}$ and bird's-eye-view AP computed over 40 recall points, for cars at IoU 0.7, and is reported overall and in 0–20, 20–40 and 40–60 m bands. We also report absolute depth error against pixel height. Because Virtual KITTI 2 clones KITTI drives, we split by raw drive so that no evaluated real scene has a rendered twin in training. The paired rendered/real scenes are used only for the diagnostic.
CARLA→KITTI is the second transfer task. It uses KITTI-matched camera intrinsics and mounting height, followed by a controlled perturbation of camera height. This tests whether the ground cue's known dependence on $h_c$ makes the method brittle, the failure CHARM3R was designed to cancel.
Three outcomes are decisive. The diagnostic must show that far-field error is dominated by 2D measurement error. With matched augmentation, the bottleneck must recover a clearly larger fraction of oracle performance in the 40–60 m band than the free-form depth head. And cue-agreement filtering must yield pseudo-labels with lower depth error by range than learned-uncertainty filtering, at equal retention rate. If the first check fails, the proposal's premise is wrong. If only the third holds, the contribution narrows to a label-filtering method for monocular domain adaptation, which is still worth publishing.
References
- Adapting Control Policies from Simulation to Reality Using a Pairwise Loss
- Pixel-Attentive Policy Gradient for Multi-Fingered Grasping in Cluttered Scenes
- Habitat: A Platform for Embodied AI Research
- CTS: Sim-to-Real Unsupervised Domain Adaptation on 3D Detection
- Transfer Learning from Simulated to Real Scenes for Monocular 3D Object Detection
- Aug3D-RPN: Improving Monocular 3D Object Detection by Synthetic Images with Virtual Depth
- Monocular 3D Object Detection via Feature Domain Adaptation
- Categorical Depth Distribution Network for Monocular 3D Object Detection
- Revisiting Depth-guided Methods for Monocular 3D Object Detection by Hierarchical Balanced Depth
- AutoShape: Real-Time Shape-Aware Monocular 3D Object Detection
- Densely Constrained Depth Estimator for Monocular 3D Object Detection
- Generalizing Monocular 3D Object Detection
- LIP: Local Importance-Based Pooling
- Virtual KITTI 2
- 3DZip: Spatial-Aware Feature Diversity-Guided Token Compression for 3D Question Answering