Symmetry-Plane Supervision for Monocular 3D Detection

A camera sees a car or a pedestrian as a flat patch of pixels. From that patch alone, a small nearby object and a large distant one look the same, so monocular 3D detectors (networks that place 3D boxes around objects using one image) make most of their errors in depth. The standard remedy is to train the detector with extra prediction heads, called auxiliary tasks, that are used only during training and deleted afterwards. These heads push the shared features to encode geometry. Current auxiliary targets are either sparse points on the outside of the object, such as the projected corners of its 3D box, or depth values borrowed from LiDAR (a laser range sensor). This proposal uses a different target. Almost every road object is mirror-symmetric about a vertical plane through its heading. That plane can be computed exactly from the 3D box labels a dataset already has, so no new annotation is needed. We train each pixel inside an object to predict two things: which way it lies from the image of that mirror plane, and how far along its own viewing ray the plane is. Both heads are deleted at test time, so inference is exactly as fast as the base detector.

The idea borrows from work on detecting symmetry axes in cluttered images. That field found two things. First, marking an axis as a thin set of "on-axis" pixels is brittle. Second, having every interior pixel point toward the axis both separates real object axes from chance symmetries in background texture and tolerates variation in object width. We bring that representation into 3D detection, where it has not been used. We also test whether it removes a known weakness of corner-based auxiliary heads: they help less on small, crowded pedestrians and cyclists.

Core Claim

We claim that a dense, interior, symmetry-plane target is a better auxiliary signal for monocular depth than sparse exterior keypoints or a constant per-object depth, and that it needs only 3D box labels. Two per-pixel heads make up the target.

The first is a flux field. For a pixel $p$ inside an object's projected 3D box, $F(p)$ is the unit 2D vector pointing to the nearest point on the projected symmetry-plane polyline. That polyline runs along the top face from front-centre to rear-centre and down the visible front or rear face. The pixel also gets a binary label saying which half of the object it lies on. Background pixels are trained toward zero flux.

The second is medial-plane depth. The mirror plane is $n^\top X = c$, where $n$ is the object's lateral axis in camera coordinates and $c = n^\top X_0$ for box centre $X_0$. For a pixel with viewing ray $r(p) = K^{-1}[u, v, 1]^\top$, where $K$ is the camera intrinsic matrix, the depth at which the ray meets the plane is

$$Z_{\text{med}}(p) = \frac{c}{n^\top r(p)} .$$

This depth varies across the object with its yaw and with perspective. It is not a single number broadcast over a mask.

The expression becomes unstable when the ray nearly lies in the plane, that is, when we see the object head-on or from directly behind and $n^\top r(p) \to 0$. For that reason the depth loss is weighted by $|n^\top r(p)|$ and truncated to the box extent. Those same head-on views are where the half-side label and the flux carry the orientation information instead.

The closest existing work, and how this differs from each:

  • Dense depth from LiDAR: MonoPixel supervises object-surface depth with sparse LiDAR and learns a surface-to-centre offset. Our medial-plane depth replaces LiDAR with an analytic target derived from the box, so it applies to camera-only datasets.
  • Constant foreground depth: MonoDETR supervises a foreground depth map with object-level depth, which is constant within each object. Ours varies per pixel and encodes orientation.
  • Sparse exterior keypoints: MonoCon uses projected corners and centre heatmaps. Its authors attribute weaker gains on pedestrians and cyclists to overlapping keypoints on small, dense objects. Our target lives in the object interior. Where objects overlap, each pixel is assigned to the nearest object in depth, so no two targets compete for the same pixel.
  • Flux representation: DeepFlux supplies the flux formulation, but only for 2D skeletons with hand-drawn axes. Here the axis is the image of a metric 3D plane.
  • Symmetry as an auxiliary signal: RetinaFace is the precedent for symmetric 3D structure used this way. It reports that a 3D face-mesh branch added 0.35% AP on top of landmarks, but its structure comes from a face-specific model.

Research Program

We will write the target generator first. It converts each KITTI 3D box and the camera calibration into three per-pixel maps: the flux field, the half-side label, and the medial-plane depth, with occlusion resolved by depth order. We will check the generator geometrically. Recovering the box yaw from the ground-truth maps should be nearly exact, and recovering the centre depth from the medial depth at the projected centre ray should be exact.

Next we attach three lightweight heads to the backbone of MonoCon: flux regression with an angular loss, half-side classification, and medial depth with an L1 loss in log depth. We keep MonoCon's own training recipe. To show the idea does not depend on one architecture, we repeat the experiment with MonoDETR as the host.

The analysis has three parts, each tied to the source of a claim:

  • Small objects. MonoCon reports interference on pedestrians and cyclists, so we measure gains per class, broken down by projected object height and by the number of overlapping neighbours.
  • Depth error. MonoPixel identifies radial (depth) error as the dominant failure, so we split localization error into depth and lateral components and bin it by distance.
  • Chance background symmetry. The symmetry-detection literature (Park et al., InnerSpec) shows that repetitive background produces valid but meaningless symmetry axes. We therefore score each validation image by the strength of chance background symmetry, using a classical mirror-symmetry response such as HMSC restricted to non-object pixels. We then test whether false positives and depth errors concentrate in high-scoring images, and whether the zero-background-flux term reduces them.

Baselines

Do nothing. The naive reference is the MonoCon backbone and 3D heads trained with no auxiliary heads. This establishes how much any auxiliary signal helps.

MonoCon. The strongest camera-only competitor on this exact question is MonoCon with its full auxiliary set (MonoCon). It is also label-only and inference-free, which is the regime we target. The file reports only a relative figure: a 1.44% absolute AP gain on KITTI cars over GUPNet, at 38.7 FPS. No reported absolute KITTI AP for MonoCon was found on Lacuna, so we will rerun it ourselves.

MonoDETR. MonoDETR reports a 2.53% AP3D margin over the second-best method on the KITTI moderate car test set. It is the strongest depth-guided model in the file and the second host.

MonoPixel. MonoPixel is a LiDAR-using upper reference. We include it to ask whether box-derived medial depth recovers most of the benefit of LiDAR surface depth.

Fair comparison. All models use identical data splits, backbone, input resolution and training epochs. Every variant, ours included, gets the same small hyperparameter sweep over auxiliary loss weight, and we report the mean and spread over three seeds.

Ablations isolating the key ingredient:

  • MonoCon heads plus our heads, versus our heads replacing MonoCon's, to see whether the signals add up or overlap.
  • Medial-plane depth versus depth of the visible box face along the ray, versus MonoDETR-style constant centre depth. This tests whether the symmetry plane specifically matters or whether any dense depth derived from the box suffices.
  • Flux regression versus binary on-axis classification, since DeepFlux argues the latter gives thick, fragmented maps.
  • With versus without the background zero-flux term.

What counts as winning. Beating MonoCon means a moderate-difficulty AP3D gain on pedestrians and cyclists larger than the spread across seeds, with no loss on cars. It also means a reduction in depth error that grows with distance.

Why not just…?

Why not just use MonoCon's corners, since the symmetry plane is already determined by the box? In terms of information content, it is. Our bet is about representation, not new information: dense interior votes versus sparse exterior points that overlap. This is a real threat. If "MonoCon plus ours" does not beat MonoCon on small or crowded objects, the representation argument fails. The per-class, per-overlap breakdown settles it.

Why not just supervise dense depth from LiDAR, or run an off-the-shelf depth network? MonoPixel reports that cascaded depth-then-detect pipelines take 400 ms or more per frame, and that general-purpose depth models leave large foreground errors. LiDAR supervision also excludes camera-only data. We include MonoPixel directly as a reference. If box-derived medial depth closes most of the gap to LiDAR, the method matters for datasets without LiDAR.

Why not just use instance masks, or depth of the box surface? KITTI lacks instance masks, and masks carry no orientation. Barron et al. show that foreshortening cannot be resolved from outlines alone. The box-surface-depth ablation is the more serious alternative: it is equally free and also dense. If it matches medial-plane depth, we will report that the symmetry framing adds nothing and that dense box-derived depth is the finding.

Why not just fix small-object interference with loss reweighting? Reweighting changes how strongly the heads compete but not where their targets sit. We will apply the same loss-weight sweep to MonoCon, so any remaining gap is not a matter of tuning.

Proposed Experiments

Primary benchmark. KITTI 3D detection (KITTI), reporting AP3D and bird's-eye-view AP at 40 recall points, with IoU 0.7 for cars and 0.5 for pedestrians and cyclists, across Easy, Moderate and Hard. The validation split is used for ablations and the test server for the final models.

Secondary benchmarks. nuScenes front-camera mAP and average translation error test transfer. MonoPixel claims reduced translation error for pedestrians and cyclists but gives no number, so no reported result was found on Lacuna for that baseline value. Waymo Level 1 at IoU 0.7 offers a harder generalization check against CaDDN's reported 5.03 mAP (CaDDN). That comparison is only indicative, because CaDDN uses LiDAR depth supervision and keeps its projection step at inference.

Decisive checks. Four results would decide the proposal:

  • The small-object gain over MonoCon increases with the number of overlapping neighbours. This is the predicted signature of removing keypoint interference.
  • Depth-error reduction grows with distance, while lateral error stays flat.
  • Medial-plane depth beats box-face depth.
  • Where chance background symmetry is strong, false positives fall relative to MonoCon.

Run-to-run safeguards. Throughput is measured with the auxiliary heads removed and must match the host network's. If the first and third checks both fail, we will conclude that dense box-derived targets, not symmetry, drive any gain, and we will report that result.