Methods that take a cluttered image apart layer by layer work well. They find the frontmost object, account for it, and repeat on what remains, and on heavily occluded scenes this beats predicting everything at once. The cost is paid every time the model runs: several passes, often diffusion sampling or generative inpainting, and errors that compound from one layer to the next. Fast monocular 3D object detectors, which place 3D boxes around cars and pedestrians from a single camera image, take the opposite approach. They predict all objects in one forward pass and improve the shared feature extractor with "auxiliary heads": extra prediction branches used only during training and deleted before deployment. This note proposes joining the two. We would run layer-by-layer peeling only as a training-time auxiliary task, then delete it, leaving a detector that is exactly as fast as before but whose features were trained to separate occluders from the objects they hide.
The opportunity is sharpest where the two literatures fail in complementary ways. Recovering depth from one image is underdetermined: a small near object and a large far one can project identically, and the error along the camera ray dominates 3D detection failures, as the MonoPixel analysis found (Boosting Monocular 3D Object Detection with Object-Centric Auxiliary Depth Supervision). The best auxiliary-head detector, MonoCon, reports that its gains shrink on small, clustered objects such as pedestrians and cyclists, because their projected keypoint targets overlap and interfere (Learning Auxiliary Monocular Contexts Helps Monocular 3D Object Detection). Avoiding that kind of mutual interference is exactly what peeling was designed for.
Core Claim
Our claim is that a lightweight peeling head improves a single-pass monocular 3D detector on occluded and clustered objects, at zero inference cost. The head predicts the geometry of occlusion layer $i$ while being told which image regions are already taken by layers $1$ through $i-1$. Training this way pushes the shared backbone to encode occluded objects separately from their occluders.
The supervision costs nothing extra. Following the observation in FD3D that projecting ground-truth 3D boxes into the image yields clean foreground depth maps without LiDAR, we derive all peeling targets from the 3D box annotations KITTI and nuScenes already provide. Those targets are the occlusion layer of each object, the amodal footprint (the full projected extent, including hidden parts), and the cumulative mask of everything in front. No synthetic compositing is needed, unlike the cut-and-paste ground truth used for amodal depth in Amodal depth anything.
How Layers Come From Boxes
For each training image we project every annotated 3D box into the image and take the convex hull of its eight projected corners as the object's amodal footprint. We then add a directed edge from A to B whenever the footprints of A and B overlap and A's nearest box surface lies in front of B's along the shared camera rays. The resulting occlusion graph has no cycles once we tie-break by box-center depth. An object's layer index is the length of the longest chain of occluders in front of it, so layer 1 holds the unoccluded objects.
The cumulative mask for layer $i$ is the union of footprints of all objects with layer less than $i$. This is the same conditioning signal that Sequential Amodal Segmentation via Cumulative Occlusion Learning used to keep 27.6 AP at the fourth occlusion layer, where simultaneous prediction fell to 12.4 AP. Here it is computed exactly from 3D labels rather than predicted.
Cuboid footprints are coarse, and unlabeled objects (KITTI's "don't care" regions) are missing from the graph. We treat both as known label noise and mask "don't care" regions out of the auxiliary loss.
Research Program
The base detector is MonoCon's design: a DLA-34 backbone (a hierarchical convolutional feature extractor) with keypoint heads for box center, projected corners, offsets, dimensions, and depth. We keep all of MonoCon's heads unchanged so that any gain is attributable to the new branch.
The peeling head is a small convolutional block shared across layers. At step $i$ it receives the backbone feature map $F$ concatenated with the cumulative mask $M_{<i}$. It predicts, only for objects in layer $i$, three things:
- the amodal footprint mask;
- projected corner and center heatmaps;
- object-centric depth, using the surface-depth-plus-offset decomposition from MonoPixel.
It also predicts a "blank" output that signals no further layers, adapted from Ao et al. The training objective is
$$ \mathcal{L} = \mathcal{L}_{\text{det}} + \lambda \sum_{i=1}^{K} \mathcal{L}_{\text{peel}}\big(h(F, M_{<i}),\, Y_i\big), $$
where $\mathcal{L}_{\text{det}}$ is the unchanged detector loss, $h$ is the peeling head, $Y_i$ are the box-derived targets for layer $i$, $K$ is the maximum number of unrolled layers (we use 4), and $\lambda$ weights the auxiliary term.
During training we always feed the ground-truth cumulative mask, a practice known as teacher forcing. Because the head is deleted at test time, the cascading-error problem of inference-time peeling (Visiting the Invisible) cannot arise. The head adds negligible training cost and none at inference.
The mechanism we are betting on is a readout constraint. A shallow head can recover a layer-3 pedestrian's corners only if the backbone features at those pixels still carry that pedestrian's geometry rather than being taken over by the car in front. Single-pass heatmaps impose no such requirement; they reward whichever object dominates a location.
Following ForeSeE and MonoPixel (Task-Aware Monocular Depth Estimation for 3D Object Detection), the peeling loss covers foreground objects only. A secondary question touches the tension between that foreground-only view and the global scale anchors argued for in Depth Map Decomposition for Monocular Depth Estimation: we will test adding a final "background layer" whose target is box-plane-consistent ground depth.
Baselines
Naive reference. The do-nothing reference is the base detector with MonoCon's auxiliary heads removed. This establishes how much any auxiliary supervision buys on our splits.
MonoCon. MonoCon itself, retrained in our codebase with identical data, schedule, augmentation, and hyperparameter search budget, is the strongest directly comparable competitor. It is the exact architecture we extend and the paper that documents the clustered-object failure we target. The file reports only that MonoCon gained "1.44% absolute improvement in AP over … GUPNet" on KITTI Car while running at 38.7 FPS. No absolute AP₃D Moderate value and no pedestrian or cyclist numbers were found on Lacuna. We will therefore compare against our own retrained MonoCon, run with the same number of seeds as our method, and report both. Beating it means a Pedestrian and Cyclist Moderate AP₃D gain larger than the seed-to-seed spread, with Car not degraded and FPS unchanged.
FD3D and MonoPixel. These are the strongest existing auxiliary methods that use object-centric depth. FD3D reports a +0.66% AP₃D gain on KITTI Moderate Car and +1.3% mAP / +0.8 NDS on nuScenes over DD3D. MonoPixel reports outperforming RGB-only methods on KITTI, with no exact figure found. For reference among single-pass depth-aware detectors, MonoDTR reports 15.39% AP₃D on KITTI Car Moderate. We will add FD3D's box-derived foreground depth supervision to the same backbone as a competing auxiliary head, with an equal tuning budget.
Ablations isolating the key ingredient. The essential ablation keeps the identical peeling head and targets but feeds an empty cumulative mask. A second ablation replaces the unrolled head with a single-pass head that outputs $K$ layer-indexed heatmap channels at once. Together these tell us whether the gain comes from layer labels, from extra amodal supervision, or from conditioning on what has already been peeled.
Why not just…?
Why not just predict layer-indexed heatmaps in one pass? This is the most serious threat. Assigning overlapping pedestrians to different output channels by occlusion layer might remove keypoint interference without any notion of peeling. It is our second ablation, run at matched parameter count. If it matches the conditioned head, we will report that the benefit comes from layer-separated targets, not from sequential consumption, and the contribution narrows to "box-derived occlusion layers as cheap auxiliary targets."
Why not just use object-centric depth supervision like MonoPixel or FD3D? Those methods tell the backbone how far each visible object is. They do not tell it how to separate an object from the one occluding it. FD3D's foreground maps explicitly cover only visible projections. We run FD3D-style supervision both alone and combined with peeling. If peeling adds nothing on top of it on the high-occlusion stratum, the idea fails.
Why not just feed a foundation depth model's output to the detector? MonoDTR reports that a cascaded depth-first detector (CaDDN) needs 630 ms per frame, versus 37 ms for implicit depth. Dense depth estimators also optimize mostly background pixels, which make up over 90% of KITTI images according to ForeSeE. We compare only at equal inference cost. A foundation-depth variant is reported separately as an upper reference, not as a competitor in the real-time regime.
Why not just peel at test time, as Seeing Through Clutter does? Seeing Through Clutter chains a vision-language model, a segmenter, diffusion inpainting, and image-to-3D generation. It reports no latency compatible with driving and needs heuristic filtering of overlapping volumes. Our question is specifically whether the benefit of peeling survives once the loop is removed.
Proposed Experiments
The primary benchmark is KITTI 3D detection, reported as AP₃D and AP_BEV for Car, Pedestrian, and Cyclist. The official Easy/Moderate/Hard splits partly track occlusion. We add our own stratification of validation objects by box-derived layer index (1, 2, ≥3) and by the number of overlapping projected footprints. nuScenes mAP, NDS, and translation error give a second domain with denser crowds. We also report KITTI-Object-Depth foreground depth error, where ForeSeE reports up to 8.5% improvement over its baseline.
The decisive check is where the gain lands. The hypothesis predicts improvement concentrated in layer ≥2 objects and in Pedestrian and Cyclist, with layer-1 Car essentially unchanged. A uniform small gain would instead suggest generic regularization.
A mechanistic check freezes each trained backbone and fits a linear probe that maps backbone features to the depth and corners of layer ≥2 objects. Our model should be more probe-decodable than MonoCon on exactly those objects.
Finally, we sweep $K$ from 1 to 4. Gains should grow with $K$ on the high-occlusion stratum and saturate where few training images have deeper layers. The two ablations from the Baselines section determine whether the cumulative-mask conditioning, rather than layer labels alone, carries the effect.
References
- Boosting Monocular 3D Object Detection With Object-Centric Auxiliary Depth Supervision
- Learning Auxiliary Monocular Contexts Helps Monocular 3D Object Detection
- FD3D: Exploiting Foreground Depth Map for Feature-Supervised Monocular 3D Object Detection
- Amodal depth anything: Amodal depth estimation in the wild
- Sequential Amodal Segmentation via Cumulative Occlusion Learning
- Visiting the Invisible: Layer-by-Layer Completed Scene Decomposition
- Task-Aware Monocular Depth Estimation for 3D Object Detection
- Depth Map Decomposition for Monocular Depth Estimation
- MonoDTR: Monocular 3D Object Detection with Depth-Aware Transformer
- Seeing Through Clutter: Structured 3D Scene Reconstruction via Iterative Object Removal