Box-Anchored View Augmentation for Monocular 3D Detection

A monocular 3D detector finds cars and pedestrians in a single camera image and reports each one's position, size and heading in meters. Diagnostic studies show the hard part is where the object is: its distance from the camera, and the image point onto which its true 3D center falls. That point usually differs from the center of the object's 2D box because of perspective. One obvious way to teach these quantities is to synthesize new training images from shifted camera positions. A new viewpoint puts the same objects at new depths and image locations, and the 3D labels can be moved exactly. In practice, this kind of synthesis warps pixels with a depth map from an off-the-shelf monocular depth estimator, which predicts distance from one image. That depth is wrong by an unknown scale and is blurred at object edges. The pixels then move by one amount and the labels by another, so the augmentation teaches the detector a systematic offset between appearance and position.

We propose to move each annotated object's pixels using depth computed from its own ground-truth 3D box rather than from the depth estimator. The projected 3D center in the new label then lands exactly on the warped object by construction. The external depth model is used only for the background. Holes opened behind moved objects (disocclusions, i.e. surfaces hidden in the source view that become visible after the shift) are masked or filled by inpainting. The claim is narrow and testable: the benefit of novel-view augmentation for monocular 3D detection is limited by how well image and label stay aligned, not by how realistic the image looks. Anchoring object warps to the labels removes the error that the scale ambiguity of monocular depth would otherwise inject.

Core Claim

For a lateral camera shift of $b$ meters, a point at true depth $Z$ moves in the image by $\Delta u = f b / Z$, where $f$ is the focal length in pixels; this follows directly from the pinhole projection. If the warp instead uses an estimated depth $\hat Z = Z(1+\epsilon)$ with relative error $\epsilon$, the object's pixels land off by about

$$\delta u \approx \frac{f\, b\, \epsilon}{Z},$$

while the label's projected 3D center is moved with the exact geometry. With KITTI-like optics ($f \approx 720$ px), a one-meter shift, a car at 10 m and a 10% depth error, that is about 7 pixels of image–label disagreement. This lands on the quantity MonoDLE identified as decisive. Delving Into Localization Errors for Monocular 3D Object Detection replaced predicted depth and projected center with ground truth and lifted car AP from about 11% to nearly 79%. That result motivates this proposal: small systematic center errors are costly. Forward camera shifts fail similarly, because the depth error rescales the warped object's apparent size, and apparent size is the main cue for distance. Box-anchored warping sets $\epsilon = 0$ on labeled object pixels. The relation also predicts where the benefit concentrates: on near objects and large shifts, which the experiments check.

Research Program

We build an offline augmentation pipeline for single-image training data. For each labeled object, we render a depth map of its ground-truth 3D box, the depth of the visible box faces at each pixel, inside its instance region. Instance regions come from an off-the-shelf segmenter, refined by intersecting with the projected box. This box depth is coarse on curved bodywork. We therefore fit a per-object scale and shift of the monocular depth estimate to the box depth and keep the estimator's fine shape. The object's mean depth and projected center are thus pinned to the label, while the local surface relief comes from the depth prior. For the background, the monocular estimate is converted to metric depth by a global scale-and-shift fit. On KITTI the fit uses the sparse LiDAR points. Without LiDAR, it uses the box depths of all labeled objects, so the labels act as the metric anchor that resolves the ambiguity.

We then forward-warp object and background layers separately to a sampled camera pose. The poses are lateral shifts up to 1.5 m and forward or backward shifts up to 5 m, with intrinsics fixed and the ground plane kept. The two layers are composited by depth order, and the labels are reprojected exactly. Disoccluded holes are handled in two arms: masking them out of the loss, and filling them with a pretrained image inpainter.

Some surfaces newly exposed by the shift are never seen in the source image, such as a car's unseen side face or the hidden part of an occluded car. These cannot be warped. We cap the shift so the change in viewing angle of each object stays small. We also recompute truncation and occlusion levels after the shift and drop labels whose newly visible area exceeds a threshold, rather than ask the inpainter to invent object geometry.

The main analysis tool is an alignment metric. On KITTI frames with LiDAR, we warp the source image with each depth source, then measure the pixel offset between the reprojected LiDAR points on each object and that object's warped pixels. For each method we report the offset as a function of distance and shift size. This tests the $\delta u$ relation directly before any detector is trained.

Baselines

The naive reference is the detector trained with its standard recipe and no view augmentation. We use MonoDLE as described in Delving Into Localization Errors for Monocular 3D Object Detection. It directly regresses the projected 3D center, down-weights objects beyond 60 m, and uses a 3D-IoU-oriented size loss. It reports 17.23% $AP_{3D}$ on KITTI test (Car, Easy) at 25 FPS. We choose it because it is the cleanest center-based detector, so any change in center alignment shows up directly.

The strongest detector-side competitor is MonoDETR: Depth-guided Transformer for Monocular 3D Object Detection. It replaces local center features with depth-guided attention over the whole image. The file reports it as +2.53% $AP_{3D}$ on KITTI test (Moderate Car) over the second-best method, but gives no absolute value: no reported result found on Lacuna for MonoDETR's absolute KITTI AP. We retrain it ourselves and report our own number. Including it tests whether the augmentation still helps a model that already reasons globally about depth. We also include Mitigating Perspective Distortion-Induced Shape Ambiguity in Image Crops. It feeds per-pixel camera ray angles to the network (KPE) and reports +5.4% $AP_{3D}$ and +16% $AP_{3D}^{75}$ for Cube R-CNN on KITTI and nuScenes. KPE addresses perspective-induced center ambiguity from the architecture side, so it is the natural rival to a data-side fix. We test augmentation with and without KPE to see whether the two gains add up.

The key ablation is identical augmentation with MDE-depth warping in place of box-anchored warping. The poses, inpainter and number of synthetic images are the same, and the depth map is globally scale-aligned to LiDAR so it gets its best chance. An oracle arm warps with LiDAR-completed depth. It shows how much of the remaining gap is depth accuracy versus layer handling.

All arms use the same KITTI train split and the same epoch count. The number of synthetic images per real image is matched. Each method gets the same hyperparameter search budget, tuned on validation only, with three seeds. Beating the baseline means two things. First, a Moderate-Car validation $AP_{3D}$ gain over the MDE-warp ablation that exceeds seed variance. Second, a lower projected-center error and depth error on near objects, where the relation predicts the gap is largest.

Why not just…?

Why not just use full generative scene augmentation, as WorldSplat does? WorldSplat: Gaussian-Centric Feed-Forward 4D Scene Generation for Autonomous Driving generates driving scenes conditioned on bird's-eye layouts and 3D boxes, inpaints disoccluded regions with a diffusion refiner, and reports +4.0 mAP for StreamPETR on nuScenes. That is a real threat. However, its object positions are generated by a model conditioned on boxes rather than fixed by geometry. The file also notes that its box queries carry no explicit model of perspective center shift. Its cost is heavy and its training setup is multi-camera, so we cannot reproduce it at matched compute. Instead we run our alignment metric on any of its renders we can obtain, or on a simpler box-conditioned inpainting stand-in. If generated objects align as well as box-anchored warps, our contribution reduces to a cheaper equivalent, and we will say so.

Why not just use scale-and-crop augmentation with relabeled depth? Resizing an image and scaling object depth inversely mimics moving closer to or farther from an object. It is cheap and geometrically exact for a single object at the image center. It produces no parallax between objects at different depths, no lateral viewpoint change, and no change in where objects sit relative to the principal point. Changing that position is exactly what Mitigating Perspective Distortion-Induced Shape Ambiguity in Image Crops proves matters for inferring shape and center. We include it as an extra arm. If it matches our forward-shift-only arm, we conclude that the value lies in lateral shifts and report that.

Why not just get a better depth model? Better depth shrinks $\epsilon$ but does not remove it. Edge blur persists: Few-Shot Novel View Synthesis Using Depth Aware 3D Gaussian Splatting documents depth errors along object silhouettes splitting surfaces in synthesized views. The single-depth-per-pixel failure on glass described in Towards Ambiguity-Free Spatial Foundation Model: Rethinking and Decoupling Depth Ambiguity persists too. The LiDAR-oracle arm sets the ceiling for "better depth." If box-anchored warping matches it without LiDAR, the labels are doing the depth model's job at no cost.

Proposed Experiments

The first check needs no detector training. On KITTI with LiDAR, we plot the measured image–label offset against $f b/Z$ times the measured depth error for MDE-warped objects. We confirm the relation holds and that box-anchored warping collapses the offset to segmentation error. If MDE-warp offsets are already near zero after scale alignment, the premise fails and we stop.

The second experiment trains MonoDLE, MonoDETR, and Cube R-CNN with and without KPE on KITTI (Are we ready for autonomous driving? The KITTI vision benchmark suite). Each runs under no augmentation, scale-and-crop, MDE-warp, box-anchored warp, and LiDAR-oracle warp. We report $AP_{3D}$ and $AP_{BEV}$ at Easy, Moderate and Hard, and $AP_{3D}^{75}$. We also report projected-center pixel error and relative depth error binned by distance. The decisive pattern is box-anchored beating MDE-warp, with the margin largest for objects under 20 m and for larger shifts, as the relation predicts. A flat margin across distances would mean the gain comes from something other than alignment.

The third experiment varies disocclusion handling (mask versus inpaint) and the label-dropping threshold for unseen surfaces. It shows whether hallucinated background or hallucinated object faces cause harm. The fourth transfers the best recipe to nuScenes front-camera images (nuScenes: A multimodal dataset for autonomous driving), using box depths alone for metric alignment. This tests whether the labels can resolve monocular scale ambiguity without LiDAR.