Visibility-Shaped Box Losses for Cooperative 3D Detection

When several cars and roadside sensors pool what they see, a car can detect objects hidden behind a truck, because another viewer can see them. This is called cooperative perception. The detectors that turn the pooled features into 3D boxes are still trained in a crude way. They penalize each box number (center, length, width, heading) separately, and they penalize every direction of error equally. For a hidden object, the errors are not equally likely in every direction. A peer sensor typically sees one or two faces of the car. The position of those faces is pinned down, while the far faces are guesswork. Small errors in how the two viewers' relative rotation is estimated also shift the peer's evidence sideways, in a known direction and by an amount that grows with distance. This proposal trains the box head to predict an error model whose shape follows this viewing geometry, and scores the box's corners under that model.

The idea in two sentences: represent each predicted box by its four ordered ground-plane corners and train it with a Gaussian likelihood whose per-corner covariance is aligned with the object's edges. The widths along each edge normal and tangent are predicted from which faces were visible to which agent and how far away that agent was. This couples all box parameters and keeps front distinguishable from back. It should also sharpen boxes where the evidence is strong and stop the network from averaging where it is weak, which matters most for long vehicles seen only by a distant peer.

Core Claim

In cooperative detection, the residual error of a box for an object hidden from the ego vehicle has a predictable structure, set by visibility and by relative-pose noise. A loss that encodes this structure should improve strict-threshold AP and heading accuracy on occluded, elongated objects. That gain should go beyond merely swapping in an existing coupled rotated-box loss. The second part is what separates this from the obvious project, and it is the part the experiments must establish.

Where this sits in prior work

Three lines of work motivate and supply the method. The first shows that decoupled regression is the wrong objective for rotated boxes. IoU Loss for 2D/3D Object Detection builds an exact differentiable rotated IoU and reports a 2–5% absolute mAP gain on KITTI across SECOND, PointPillars and PointRCNN. The Gaussian family maps a box to a 2D Gaussian and trains with a divergence between Gaussians, which makes angular penalties scale with aspect ratio: Detecting Rotated Objects as Gaussian Distributions and Its 3-D Generalization, Learning High-Precision Bounding Box for Rotated Object Detection via Kullback-Leibler Divergence and The KFIoU Loss for Rotated Object Detection. All of these are single-agent. They also constrain our design. A box-as-Gaussian is identical under a 180° rotation, so it cannot tell front from back, and it goes flat for square footprints. For this reason we do not use the box-as-Gaussian representation.

The second line concerns directional error bias. ADNet shows, for face landmarks, that weighting errors along the boundary normal separately from errors along the tangent speeds up convergence. We borrow the normal/tangent decomposition. In our case the weights are predicted per corner from observation geometry rather than fixed.

The third line concerns uncertainty under partial views. Gaussian YOLOv3 predicts a variance for each box coordinate separately. Autoregressive Uncertainty Modeling for 3D Bounding Box Prediction argues that single-view occlusion makes pointwise 3D box regression underspecified. It models the joint distribution by predicting size, then center, then heading, each over discretized bins searched with beam search, and it reports calibrated quantile boxes on COB-3D. Our head is a single forward pass with a structured continuous covariance, and it targets a multi-agent regime that paper does not study.

On the cooperative side, V2VNet, V2X-ViT, Where2comm, CoBEVT and EMIFF put their effort into how intermediate features are aligned, attended to and compressed. V2X-ViT explicitly models pose misalignment in its feature attention. None of them changes how the final boxes are supervised. That is the gap we address, but not merely by transplanting one loss into another pipeline.

Research Program

The box head predicts, for each object, four corners in the Bird's Eye View (BEV, the top-down ground-plane map where fused features live). The corners are ordered front-left, front-right, rear-right, rear-left, and a 1D vertical extent is predicted as usual. Ordered corners determine center, size and heading jointly and are not symmetric under a half-turn. For each corner $k$, let $n_k$ and $t_k$ be the unit normal and tangent of its adjacent edge in the predicted box frame. The head predicts two scalars, $\sigma_{n,k}$ and $\sigma_{t,k}$, and forms a covariance $\Sigma_k$ in the ground plane:

$$\Sigma_k = \sigma_{n,k}^2\, n_k n_k^\top + \sigma_{t,k}^2\, t_k t_k^\top + (r_k\,\sigma_\phi)^2\, u_k u_k^\top .$$

The third term is a fixed geometric prior, not a learned one. It represents error from the unknown relative-rotation noise $\sigma_\phi$ of the agent that contributed most evidence at that corner. Here $r_k$ is that agent's distance to the corner, and $u_k$ is the direction perpendicular to that agent's line of sight. A small rotation error $\delta\phi$ displaces a point at range $r$ by about $r\,\delta\phi$ sideways, which is where this term comes from.

The loss is the corner-wise Gaussian negative log-likelihood, where $r^{\text{res}}_k$ is the residual between predicted and true corner $k$:

$$\mathcal{L} = \sum_{k=1}^{4} \tfrac12\, {r^{\text{res}}_k}^{\top} \Sigma_k^{-1} r^{\text{res}}_k + \tfrac12 \log\det \Sigma_k .$$

The network predicts the $\sigma$'s from the fused features plus an explicit visibility descriptor. To build that descriptor, we cast rays from each agent's sensor origin to the four faces of the ground-truth box during training, or the predicted box at test time. This gives, per face, which agents could see it, at what range and at what grazing angle. Visible faces should earn small normal variance and occluded far faces large variance. The network learns the mapping, while the rotation-noise term tells it which part of the error it cannot remove.

To avoid the well-known failure where a likelihood loss escapes hard examples by inflating variance, we add a rotated-IoU term, via the analytical layer from IoU Loss for 2D/3D Object Detection, on the mean corners with a small weight. We also floor $\sigma$ and report calibration. The predicted covariances double as an uncertainty output, and we check them against empirical corner coverage, in the spirit of the quantile boxes in Autoregressive Uncertainty Modeling for 3D Bounding Box Prediction.

Baselines

The naive reference is the fusion model exactly as published, with its standard independent Smooth $L_1$ box regression plus heading-direction classification. We use V2X-ViT with its PointPillars encoder on V2XSet and OPV2V, and a Where2comm-style model on DAIR-V2X-C. For the fusion architecture, V2X-ViT reports a 21.2% AP gain over single-vehicle perception on V2XSet and a 7.3–8.8% margin over V2VNet and DiscoNet under noise. It also reports a 5–10% drop under 0.2 m pose error and 100 ms delay, and no absolute AP value for it appears in the file. Where2comm reports improving the prior best on DAIR-V2X by 7.7% AP. The reference figures quoted in the file for DAIR-V2X-C are late fusion at 41.90 AP and early fusion at 50.03 AP (DAIR-V2X). For OPV2V, the file quotes 0.734 [email protected] for the PIXOR-based pipeline on default towns versus 0.406 without fusion (OPV2V).

The strongest competitor is the same fusion model with the box head's regression replaced by an existing coupled rotated-box loss. We run two versions. One uses KFIoU plus a direction classifier, since KFIoU is the strongest Gaussian loss on DOTA-v1.0 (77.35% mAP) and gives +2.43 3D mAP over the PointPillars baseline on KITTI (The KFIoU Loss for Rotated Object Detection). The other uses the exact rotated 3D IoU/GIoU loss. For any cooperative benchmark, no reported result was found on Lacuna for KFIoU, KLD or rotated IoU losses. We produce these numbers ourselves, and they are the real bar.

We add two ablations that isolate the key ingredient. The first is an isotropic corner NLL with one learned variance per corner, which keeps coupling and heading but removes the edge-aligned shape. The second is an independent per-parameter heteroscedastic head in the manner of Gaussian YOLOv3. A third ablation removes only the ray-cast visibility descriptor and the rotation-noise prior, leaving the anisotropy fully learned.

Fairness is enforced by holding everything but the head's loss fixed: identical backbone, fusion module, data splits, pose-noise injection, epochs and augmentation. Every variant gets the same tuning budget, a fixed sweep over learning rate and loss weight, and we report three seeds. We count the proposal as beating the strongest competitor only if it improves [email protected] on the occluded-and-elongated stratum by more than the seed-to-seed spread. It must also not lose overall AP.

Why not just…?

Why not just swap KFIoU or rotated IoU into the cooperative head? This is the real threat. The single-agent gains are consistent enough that such a swap will very likely help. If it captures the whole benefit, then the visibility-shaped covariance is not worth its complexity, and we will say so. Our argument for why it should not capture everything is specific. Neither loss knows that an occluded object's far faces are unobserved. Both push the network to hit every corner equally hard, and on unobservable edges this trains the model toward averaging. The Gaussian versions additionally cannot fix heading without a separate classifier. The stratified comparison below settles the question.

Why not let feature-level attention absorb misalignment, as V2X-ViT's multi-scale window attention does? Attention can move features into rough alignment. However, V2X-ViT still loses 5–10% under 0.2 m pose error and 100 ms delay. In the single-agent setting, IoU Loss for 2D/3D Object Detection shows that output supervision matters even with strong backbones. We build on top of V2X-ViT rather than replacing it, so any gain is additive by construction. The pose-noise sweep tests whether the gain grows as residual misalignment grows.

Why not use the autoregressive uncertainty head, which already handles partial views? It gives calibrated joint uncertainty, but it relies on beam search over 512 bins per parameter, which is hard to fit into a driving loop (Autoregressive Uncertainty Modeling for 3D Bounding Box Prediction). We will include it as a calibration and accuracy comparison on the occluded stratum and report latency alongside it. If it is clearly better at matched latency, that argues against our head.

Why not simply learn per-coordinate variances? That is the Gaussian YOLOv3 ablation. It cannot express that uncertainty lies along a particular edge direction, because the corners along an unseen edge are correlated through heading and length. It is a direct test of whether coupling the covariance to geometry matters.

Proposed Experiments

We train and evaluate on OPV2V and V2XSet (simulated, with controllable pose noise and delay) and on DAIR-V2X-C (real vehicle–infrastructure data). We report overall [email protected] and [email protected], then stratify objects along two axes. The first is ego-visible versus peer-only-visible, using the ego sensor's point count inside the box. The second is aspect ratio in terciles. Simulated fleets are dominated by cars, so the long-vehicle stratum may be small, and we will report its size and confidence intervals. We also report mean heading error, the rate of 180° heading flips, and empirical coverage of the predicted corner covariances at nominal 50%, 80% and 95% levels.

The first decisive check is the peer-only, high-aspect-ratio cell at [email protected]. The proposal must beat both the KFIoU and rotated-IoU variants there beyond seed noise. The second is a pose-noise sweep on V2XSet, with rotation noise from zero upward. If the rotation-noise term is doing real work, the gap to the strongest competitor should widen with noise and vanish at zero noise. A flat gap would mean the prior is decorative. The third is the flip rate. Ordered corners should match or beat a Gaussian loss with a direction classifier without any classifier. The fourth is calibration. Corner coverage should be close to nominal in the peer-only stratum, which is where an isotropic variance head is expected to be overconfident along unseen edges and underconfident across seen ones.

If the first check fails, the conclusion is that swapping in a coupled loss is sufficient for cooperative detection, which is a useful negative result in its own right. The visibility-shaped covariance would then be dropped.