Physical Exclusion Along Camera Rays for Camera-Only Collaborative Perception

A single camera cannot tell whether a car is small and near or large and far. Every detected pixel defines a line of sight, and the car could sit anywhere along it. Camera-only cooperative driving systems, where several vehicles and roadside units share what they see over a wireless link, try to settle this by intersecting lines of sight from different viewpoints. That works when two agents see the same object and their positions are known precisely. It fails for objects only one camera sees, and when position errors or transmission delays stop the rays from meeting. In those cases the system produces "ghost" cars smeared along the ray.

We propose to use a fact every agent already knows and none currently exploits: solid things cannot overlap. A hypothesized car at depth $d$ along a ray is physically impossible in three cases. Its body may intersect a collaborating vehicle whose own shape and pose that vehicle broadcasts. It may intersect another confirmed object. Or it may occupy space that another agent has just seen through, meaning that agent's line of sight passed that point before hitting anything. We turn this into a differentiable penalty ("interpenetration energy") over candidate depths. We feed the penalty back into the stage that scores 2D-derived object candidates, so impossible hypotheses are suppressed before they become boxes rather than cleaned up afterward.

Core Claim

In camera-only collaborative detection, the strongest cue for single-observer objects is often negative evidence rather than a second positive sighting. Negative evidence is either known occupancy (collaborator bodies) or known emptiness (other agents' swept free space). Encoding it as a soft, analytic volume-overlap penalty on depth hypotheses, with a tolerance margin that preserves legitimate close contact, should reduce ghost detections and depth error. The largest effect should appear exactly where ray triangulation cannot help: objects seen by one agent, and localization noise beyond what current ray-based fusion tolerates. The new element is the use of physical exclusion as a constraint on the depth distribution of each camera query inside the fusion loop. Adding an overlap loss on output boxes, which is close to what non-maximum suppression already does, is not the contribution.

Where This Sits

The closest work is RayFusion. It shares ray origin, direction, and occupancy probability among agents on top of the Sparse4D detector, and it triangulates positions where rays meet, tolerating localization offsets up to 0.6 m and latency up to 500 ms. We build directly on its ray representation; it supplies the method. We differ in using rays that do not intersect a target: a collaborator's ray that passes through a region before its first hit certifies that region as empty. We also add solid-body reasoning, which RayFusion does not have, because it treats targets as point or box queries.

RC-GeoCP anchors smeared camera features with 4D radar points and transmits features where agents disagree. It motivates the proposal by naming depth smearing as the central camera failure, but it answers with extra hardware. We instead ask how far pure geometry and physics can go. STAMP aligns heterogeneous agents into a shared top-down feature space. It documents a "bottleneck effect," where depth-noisy camera agents degrade stronger partners, and it leaves correction to learned alignment. We treat that effect as evidence that unconstrained camera features need a physical filter. Earlier learned intermediate fusion, namely V2VNet, OPV2V's attentive fusion, and Where2comm's confidence-driven communication, contains no term that penalizes two solid objects occupying the same space.

The penalty itself is borrowed from single-agent mesh work, which supplies the method and also constrains the design. Keep It SMPL approximates bodies with capsules so that overlap is analytic and differentiable. Real-time pose and shape reconstruction of two interacting hands shows that Gaussian-mixture volumes make non-interpenetration cheap enough for 30 FPS. That matters because dense collision tests scale poorly with many entities, so we represent each vehicle hypothesis as a small Gaussian mixture. The main negative lesson is the contact–penetration trade-off: overly strong overlap penalties push objects apart and create "floating" artifacts. CPF counters this with attractive springs, ContactOpt by tolerating 1–2 mm of overlap, and Layout-your-3D with a tolerant collision loss for lifted scene instances. The driving analogue is bumper-to-bumper traffic, where a hard penalty would delete real cars. We therefore adopt a tolerance margin, and we test for the analogue of floating explicitly.

The cascade literature explains why the penalty must act upstream. Accurate Localization of 3D Objects from RGB-D Data Using Segmentation Hypotheses shows that downstream 3D stages cannot recover what upstream 2D stages get wrong. Flow Matching for Probabilistic Monocular 3D Human Pose Estimation shows that deterministic lifting produces confident but wrong depth. Both point to keeping a distribution over depth and correcting it before commitment, which is what we do.

Research Program

Each camera query from the Sparse4D/RayFusion pipeline carries a class label, a ray, and a discrete depth distribution $p(d)$. Each depth bin implies a vehicle volume $V(d)$, built from a class-conditional size prior and oriented along the predicted heading. It is represented as a few Gaussians, much as bone-length priors in Weakly Supervised 3D Hand Pose Estimation via Biomechanical Constraints remove scale ambiguity.

Each agent additionally broadcasts two small packets. The first is its own body footprint and pose. The second is a coarse per-ray first-hit depth, which defines its swept free space. The first-hit depth is the only addition to what RayFusion already transmits, a few bytes per ray.

For each depth bin we compute an exclusion energy

$$E(d) = \sum_{k} \max\big(0,\; \mathrm{ov}(V(d), O_k) - \tau\big),$$

where $O_k$ ranges over collaborator bodies, other high-confidence object volumes, and free-space volumes from other agents. The function $\mathrm{ov}$ is the analytic Gaussian overlap, and $\tau$ is a tolerance margin that permits contact without penetration. The reweighted depth distribution $p'(d) \propto p(d)\exp(-\lambda E(d))$ replaces $p(d)$ before query refinement. The weight $\lambda$ controls how strongly exclusion is applied. The query's classification score is also discounted by the mass removed, which is the bidirectional feedback from 3D conflict to the 2D-derived candidate.

Because localization error makes the evidence itself uncertain, the free-space and body volumes are inflated by each agent's pose covariance. We also study a second use: when two collaborators' broadcast bodies interpenetrate after transformation into a common frame, that conflict is a direct signal of relative pose error, and minimizing it can correct misalignment.

Baselines

The naive reference is single-agent Sparse4D with no collaboration. Next to it we run camera late fusion with 3D non-maximum suppression, for which DAIR-V2X reports 26.56 AP for cooperative image-based late fusion against 12.03 AP vehicle-only.

The strongest competitor is RayFusion. It is the leading camera-only collaborative detector in the file and already targets ghost objects along rays. It reports an AP70 improvement of 8.21% on OPV2V and 3.64% on DAIR-V2X over previous leading models. The file gives no absolute AP value, so we will retrain it from its released configuration and report our own numbers. For fairness, our method is implemented as a module inside RayFusion's codebase. Both systems use the same backbone, training schedule, data splits, and noise and latency injection protocol, and both receive the same hyperparameter search budget. We also report bandwidth, so the free-space packet's cost is visible.

We additionally compare against Where2comm run in camera mode, because it is the strongest communication-aware intermediate fusion method.

Beating RayFusion means a statistically reliable AP70 gain over three seeds on OPV2V and DAIR-V2X at noise at or above 0.6 m, with no loss at zero noise. On single-observer objects specifically, it also means a clear drop in ghost rate.

Three ablations isolate the ingredients. The first applies the same energy post hoc to output boxes, which isolates the in-loop depth reweighting from mere box-overlap cleanup. The second removes the free-space term and keeps only collaborator bodies, and the third does the reverse. The fourth sets $\tau=0$ to expose the contact trade-off.

Why not just…?

Why not just run 3D non-maximum suppression or overlap-aware box clustering? This is the most serious threat to a naive version of the idea. Suppressing overlapping output boxes is already a crude non-interpenetration rule. It only compares predictions with predictions, however. It cannot use collaborator bodies or other agents' free space, and it cannot restore a correct depth once the wrong one has won. It also tends to merge genuinely adjacent cars. The post-hoc ablation, which uses the same energy but applies it to boxes, directly measures how much of our gain an output-level rule captures. If it captures nearly all of it, the in-loop claim fails, and we would say so.

Why not just rely on ray triangulation, as RayFusion already does? When two agents see an object, intersecting their rays is stronger evidence than exclusion, and we expect little gain there. We therefore stratify every result by the number of agents observing each object and by localization noise level. The proposal stands only if gains concentrate in single-observer objects and in noise above RayFusion's 0.6 m tolerance, where rays fail to meet.

Why not just give the fusion network free-space masks as an extra input and let it learn? This is plausible. ContactOpt found explicit contact optimization beat a learned refinement network, but that finding comes from hand grasps, not driving. We will train a variant that receives identical free-space and body packets as input channels, with the same budget. If it matches the explicit energy, the contribution reduces to "share negative evidence," which is still useful but narrower. We expect the explicit form to degrade more gracefully under noise levels unseen in training.

Why not just add radar? RC-GeoCP shows radar anchors work, reaching 44.55% [email protected] on V2X-Radar. Mixed fleets will still include camera-only vehicles, though, and our constraint is orthogonal to radar. On V2X-R we will add the exclusion energy to a radar-camera pipeline to check whether it helps where radar returns are missing.

Decisive Experiments

We use OPV2V and V2XSet in simulation, which provide exact ground-truth free space and body meshes, and DAIR-V2X for real vehicle–infrastructure data. Metrics are AP at IoU 0.5 and 0.7, ray-depth error of true positives, and ghost rate, defined as false positives lying within a corridor along a true object's ray. We also report the fraction of predicted objects interpenetrating collaborators or each other, and recall on close-proximity pairs (centers under 1.5 m apart), which exposes over-suppression.

Noise sweeps extend localization error from 0 to 1.5 m and latency from 0 to 500 ms. The decisive check has two parts. First, relative to RayFusion, ghost rate and depth error should fall on single-observer objects while close-pair recall stays level. Second, the in-loop version must beat the post-hoc version. If both hold, physical exclusion is doing work that neither triangulation nor output-level suppression does. If close-pair recall drops as $\tau$ shrinks, we will have reproduced the floating-versus-penetration trade-off in a new domain and can map where a good margin lies.