Free Space from Boxes: LiDAR-Free Occupancy Supervision for Monocular 3D Detection

A single camera image cannot say how far away anything is. A small car nearby and a large car far away can produce the same pixels, so a network trained to regress 3D boxes from one image gets contradictory signals and tends to latch onto brittle cues. The most effective fix so far is an auxiliary task: an extra prediction head used only during training that forces the shared image encoder to learn more about 3D structure, and is then thrown away, so inference costs nothing extra. The strongest version of this idea asks the network to predict 3D occupancy, meaning which cells of space along each camera ray are empty, filled by an object, or hidden. That version needs a LiDAR scanner at training time to know which space is empty.

We propose to build the same empty/filled/hidden supervision without any LiDAR, using only the 3D box annotations a detector already trains on, plus a ground plane fitted to the bottoms of those boxes. Each pixel's ray is marked empty from the camera until it enters an annotated box or hits the fitted ground. Cells inside the box are marked filled, and everything behind is marked unknown. The claim is that most of the benefit of LiDAR-derived occupancy comes from teaching the network where objects are not, and that boxes plus ground already carry most of that information. This matters most for pedestrians and cyclists, where existing LiDAR-free auxiliary tasks break down.

Core Claim

Two families of auxiliary regularizers currently exist for monocular 3D detection. MonoCon supervises 2D projections of box corners and offsets. It needs no LiDAR, but its authors report degradation on pedestrians and cyclists, which they attribute to crowded, overlapping projected keypoints. OccupancyM3D and MonoPixel supervise along rays and volumes and address the radial ("how far along the ray") error. MonoPixel shows this radial error is the dominant localization bottleneck. Both of these methods require synchronized LiDAR. MonoGround is the closest LiDAR-free relative: it samples the bottom faces of boxes into a dense "grounded depth" map. That map is still surface depth, though. It tells the network where the first surface is, never that the space in front of it is empty or that the object has volume.

Our claim is specific. A free-space and occupancy target computed only from boxes and a box-fitted ground plane, trained with OccupancyM3D's architecture unchanged, will recover a large fraction of OccupancyM3D's gain over its CaDDN starting point. It will beat MonoGround-style surface-depth supervision from the same annotations. It will help pedestrians and cyclists rather than hurting them, because a volumetric target does not depend on a visible ground-contact point or on separable corner keypoints. A secondary claim is that box-derived labels complement LiDAR. LiDAR only sees the front of an object and leaves the interior "unknown", whereas a box fills the whole volume. The union of the two label sources should therefore beat LiDAR alone.

How the Labels Are Built

For each training image, we fit a local ground plane by least squares to the bottom-face corners of all annotated boxes. When fewer than three boxes are present, we fall back to the dataset's nominal camera height and zero pitch. For a ray through pixel $u$, let $t_b(u)$ be the distance at which it first enters any annotated box and $t_g(u)$ the distance at which it meets the ground plane. The label for a cell at distance $t$ along that ray is

$$ y(u,t)=\begin{cases}\text{free} & t<\min(t_b,t_g)\ \text{and cell lies below height } h_{\max}\text{ above ground},\\ \text{occupied} & \text{cell inside an annotated box},\\ \text{unknown} & \text{otherwise},\end{cases} $$

where $h_{\max}$ is a height cap of a few meters. The cap is the main defense against the method's central weakness. Boxes only cover labeled classes, so a ray that passes through an unlabeled tree, pole, or building would be wrongly called free. The cap restricts "free" to the near-ground corridor where annotated road users live. KITTI's "don't care" regions and all space above the cap become unknown and contribute no loss. Rays that pass above the horizon without hitting a box or the ground are also unknown. The labels feed OccupancyM3D's two grids unchanged: a perspective-aligned frustum grid for near range and a metric voxel grid for far range. They are computed once offline, and no ray-tracing against point clouds is needed.

Research Program

We will fork the OccupancyM3D codebase (SPengLiang/OccupancyM3D) and replace its LiDAR-derived occupancy labels with the box-and-ground labels above. The backbone, heads, loss weights, and schedule stay fixed, so the only change is where the supervision comes from.

The first analysis is label fidelity. On KITTI, where LiDAR exists, we will measure how often box-derived "free" cells are contradicted by LiDAR returns, broken down by distance and by height above ground. This tells us how much label noise the height cap leaves.

The second analysis is the training comparison across label sources: none, box-derived, LiDAR-derived, and their union.

The third is a per-class breakdown, with Pedestrian and Cyclist as first-class results rather than an afterthought. If the method's value is concentrated anywhere, it should be there.

The fourth is a transfer check from KITTI to nuScenes. Geometry-aware augmentation reports an 89% relative mAP gain under that shift. Multi-source auxiliary tasks for depth found that semantic auxiliary tasks gave no benefit on KITTI. Together these suggest geometric auxiliary targets may transfer better than semantic ones, and we test this directly for our target.

Baselines

The naive reference point is OccupancyM3D's own detector with the occupancy branch removed. That detector is essentially its CaDDN-style starting point, which the OccupancyM3D paper reports at 13.41% moderate-Car AP3D on the KITTI test set. Any auxiliary target must first beat this.

The strongest existing method, and the upper reference, is LiDAR-supervised OccupancyM3D. It reports 17.02% moderate-Car AP3D on the KITTI test set in Learning Occupancy for Monocular 3D Object Detection. Its README gives validation AP3D at IoU 0.7 of 26.87/19.96/17.15 (easy/moderate/hard) and a repository re-run of 28.64/19.84/17.77. The same paper reports 10.61% mAP at IoU 0.7 on Waymo. We will retrain it ourselves with our seeds and schedule so all rows share compute. Matching the re-run's 19.84 moderate validation AP3D is the bar for "reproduced". Our success criterion is ordered. Box-derived labels should close at least half of the gap between the no-occupancy detector and LiDAR OccupancyM3D. The union of box and LiDAR labels should exceed LiDAR alone.

The key ablation isolates free space. On the identical architecture we will train with MonoGround-style dense grounded surface depth (MonoGround reports 18.69% moderate-Car validation AP3D in its own architecture). This target is built from exactly the same boxes and ground plane as ours, but it supervises only the first surface. If it matches our target, the free/occupied formulation adds nothing.

We will also run MonoCon-style projected-keypoint heads on this backbone. MonoCon reports +1.44 AP over GUPNet on Car, and no absolute pedestrian or cyclist number is given in the file, so we will measure it ourselves. The same holds for OBMO's frustum-shifted soft labels (OBMO), which we will add on top of our target to test whether the two are complementary.

All methods get the same data splits, the same number of epochs, three seeds, and an equal small sweep over the auxiliary loss weight. No reported result was found on Lacuna for pedestrian or cyclist AP of any of these methods, so those comparisons will rest entirely on our own runs.

Why not just…?

Why not just use MonoGround's grounded depth map, which comes from the same boxes? This is the real threat. Both targets come free from annotations, and if surface depth carries the same information our method is redundant. The difference is that a surface target never tells the network that the space in front of an object is empty or that the object has depth behind its visible face. OccupancyM3D argues that exactly this empty-space signal suppresses features smeared along the ray. The matched ablation above settles the question directly. We would drop the claim if surface depth matched us within seed variance, especially on pedestrians, where MonoGround itself reports its small-footprint weakness.

Why not just use LiDAR, since most driving datasets have it? Where LiDAR exists, the union experiment is the relevant test. Box interiors supply occupancy that LiDAR marks unknown, so the union may still help. Where LiDAR is absent, as in camera-only fleets or datasets labeled from other sensors, LiDAR-based methods such as OccupancyM3D and MonoPixel simply cannot train. If box-only labels recover little of the LiDAR gain, the honest conclusion is that LiDAR's value lies in covering unlabeled structure, and the label-fidelity analysis will show where.

Why not just use geometry-aware augmentation or OBMO soft labels, which are cheaper? Both address the same ambiguity at the input or label level, and both report large gains (geometry-aware augmentation: about 6% AP on KITTI validation; OBMO: 1.18–9.36% 3D gains). Neither adds a volumetric feature constraint, so we expect them to stack with our target rather than replace it. We will run each alone and in combination with our labels. If our target adds nothing on top of them, it is not worth its preprocessing cost.

Why not just use a larger pretrained depth backbone? Towards Ambiguity-Free Spatial Foundation Model reports that scaled depth models lock onto a single depth hypothesis rather than resolving ambiguity. Cascaded depth-then-detect pipelines like CaDDN run at 630 ms versus 37 ms for auxiliary-regularized MonoDTR (MonoDTR). Our auxiliary head is discarded at test time, so it can be applied to any backbone. We will report one run with a stronger backbone to check that the gain persists.

Decisive Checks

The experiments run on the KITTI 3D benchmark ($AP_{R40}$ at IoU 0.7 for Car and 0.5 for Pedestrian and Cyclist, validation and test), the Waymo Open Dataset (Level 1 and Level 2 mAP), and zero-shot KITTI-to-nuScenes transfer (mAP and translation error).

Three outcomes decide the proposal. First, box-derived occupancy must beat surface-depth supervision built from the same boxes on the same architecture, with the margin largest on Pedestrian and Cyclist. Second, it must recover a substantial share of LiDAR OccupancyM3D's gain over its no-occupancy starting point, and the box-plus-LiDAR union must exceed LiDAR alone. Third, the label-fidelity study must show that free-space errors below the height cap are rare. If they are frequent, the gains, or their absence, should be read as a label-noise problem rather than a failure of the idea.

A sweep over the height cap from ground-only to unbounded tests the mechanism directly. Performance should peak at a moderate cap and fall as the cap grows and unlabeled structure is mislabeled as free.