Visibility-Correct Viewpoint Augmentation for Camera-Based Vehicle–Infrastructure Cooperation

Roadside cameras can see what a car cannot: the pedestrian behind a bus, the vehicle around a corner. Cooperative perception, where cars and roadside units share what they observe, helps most for exactly these hidden objects. The camera-only version of this setup is weak, though. It is also the version that cheap deployments would actually use. Each roadside camera sits at its own height and tilt. A detector trained at a few intersections tends to memorize those mounting geometries and breaks at a new one. The standard fix in single-car vision is to warp training images so they look as if taken from other camera poses. That warp is a homography, a 2D projective transform that is only geometrically exact for camera rotations or for points on the ground plane. When the camera is raised or lowered, it does not change which objects block which. In a cooperative system, those occlusion relationships are the very thing the fusion model has to learn.

We propose to synthesize new viewpoints by warping each training image with its own LiDAR depth. Pixels are drawn in near-to-far order, so nearer surfaces correctly hide farther ones. Regions the original camera never saw are left as explicit "unknown" holes rather than filled in. The detection targets are then recomputed so that each synthetic view is only asked to find the objects it could actually see. Objects that become hidden in the synthetic view are kept only in the cooperative target, which trains the fusion model to obtain them from the other agent.

The idea borrows a pattern from hybrid marker–markerless tracking. A precise but expensive sensor (there, motion capture; here, LiDAR) is used only at training time to label and supervise a cheaper sensor that must work alone at test time. The claim is testable on DAIR-V2X by holding out entire intersections.

Core Claim

Camera-based cooperative detectors overfit to mounting geometry, and homography augmentation fixes the geometry only partly, because it gets occlusion wrong under camera translation. Depth-based warping with visibility-corrected labels should close more of the gap to unseen roadside mounts. The gain should be concentrated on objects hidden from the vehicle's own camera, since that is where wrong visibility labels do the most damage. The new element is not viewpoint augmentation itself. It is making the augmentation agree with occlusion, and using that agreement as a training signal for cooperative fusion.

Prior Work and Where This Sits

Motivation from cooperative perception. Several papers establish that sharing views recovers occluded objects. V2VNet shares compressed bird's-eye-view (BEV, a top-down map of the scene) features between cars and reports over 60% better detection of occluded objects. DAIR-V2X releases paired vehicle and roadside LiDAR plus camera data from 28 Beijing intersections. Its camera-only numbers show both the value and the weakness of the camera path: vehicle-only cameras reach 12.03 AP, and late fusion with the roadside camera reaches 26.56 AP, against 41.90 AP for LiDAR late fusion.

Motivation from heterogeneous fusion. HEAL adds new agent types by freezing a shared fusion network and training only the newcomer's encoder. UniTrans learns a universal translator between sensor feature spaces and reports 0.553 AP on DAIR-V2X. UniTrans also notes large drops when translating across very different viewpoints and fields of view. Both papers treat a new viewpoint as a new feature space to align after the fact. We instead try to make the camera encoder itself insensitive to mounting geometry, so that a new roadside unit needs no new encoder.

Method source: viewpoint augmentation. DG-BEV supplies the main technique we reuse and extend. It warps images with homographies to simulate new pitch, roll, yaw and height, and it predicts depth in a form that does not depend on focal length. This raises nuScenes→Waymo transfer from 0.040 to 0.297 mAP. FS-Depth simulates focal-length changes by cropping and resizing. Both are single-vehicle methods, and neither models how occlusion changes when the camera moves. The research file names that limitation directly: re-projection from sparse data creates artificial disocclusions and missing regions. We turn this limitation into the core design decision: holes stay holes, and labels follow visibility.

Method source: point-cloud re-projection. Re-projecting point clouds into simulated cameras has precedent. SEIL does it for robot imitation learning, and MVDI for depth-based action recognition. Neither handles occlusion-dependent labels or multiple agents.

Methodological lens: privileged sensors as supervisors. 3D-POP uses Vicon motion-capture markers to automatically annotate a markerless model of freely moving birds. That markerless model then fills marker dropouts with 9.2 mm error, versus 52.1 mm for linear interpolation. We use roadside and vehicle LiDAR in the same role, as a training-time annotator of what each camera can see.

Constraint: camera fallback. SiMO shows that jointly trained camera–LiDAR models collapse to near 0% when LiDAR fails. SiMO keeps 80.81% AP@30 on cameras alone, but its camera path is still limited by monocular depth. This constrains our design: the camera encoder must be trained to stand on its own, which is exactly the regime we target.

Research Program

Depth-warped views. For each training frame from each agent, we project the agent's LiDAR into its camera to get a sparse depth map, densify it with a depth-completion step in the style of SFD, and sample a virtual camera pose. Around roadside mounts, the sampled perturbations are a few meters of height and roughly ten degrees of pitch and yaw. Around vehicle mounts, they are smaller. We then forward-warp the image into the virtual pose, drawing nearer pixels over farther ones. Pixels with no source are marked with a validity channel, and the encoder receives that channel as input. Depth targets follow DG-BEV's focal-independent form so that intrinsics can be varied at the same time. We deliberately do not span the full vehicle-to-roadside height gap, because warps that large are mostly holes. We test this assumption with a widened-range variant.

Visibility-corrected targets. We cast rays from the virtual camera through the warped depth to count how many LiDAR points on each annotated 3D box remain visible. A box below a visibility threshold is removed from that agent's single-view detection loss. It is kept in the cooperative loss whenever the other agent sees it. As a result, the fusion module is trained on many synthetic cases where "the roadside unit must supply this object." Without the correction, such cases would appear as contradictory labels telling the camera to detect something it cannot see.

Integration. The augmentation is model-agnostic. We apply it to two cooperative backbones: a late-fusion camera pipeline matching the DAIR-V2X benchmark, and an intermediate-fusion BEV pipeline using HEAL's shared-feature design. This lets us see whether the effect survives a change of fusion style. At test time no LiDAR is used anywhere.

Baselines

Naive reference. The do-nothing baseline is the same cooperative camera detector trained with standard 2D augmentation only. Its anchor is DAIR-V2X's reported 26.56 AP for camera late fusion (12.03 AP vehicle-only). We retrain it on our leave-intersections-out split so that all methods share the split.

Strongest augmentation competitor. The closest augmentation method is DG-BEV's dynamic perspective augmentation together with its focal-independent depth. We use the same perturbation distribution as our method, so the only difference is homography versus depth warp. DG-BEV reports 0.297 mAP for nuScenes→Waymo, but no reported result was found on Lacuna for DG-BEV on DAIR-V2X or in any cooperative setting.

Strongest alignment competitor. This is HEAL-style backward alignment: freeze the fusion network and train a new encoder for the held-out intersection using a small labeled budget from that site. No reported camera-only held-out-intersection number was found on Lacuna for HEAL. UniTrans's 0.553 AP on DAIR-V2X is the strongest reported DAIR-V2X detection number in the file, but it comes from a mixed-modality setting. We cite it as context, not as a like-for-like target.

Ablations isolating the key ingredient. Three ablations isolate what matters. The first is depth warping without visibility relabeling, which tests whether correct labels matter beyond correct pixels. The second fills holes by simple inpainting instead of masking, which tests whether honest "unknown" regions matter. The third widens the warp range, which tests where the augmentation stops helping.

Fairness. All arms share the same backbone, the same training steps and the same hyperparameter search budget. The HEAL arm gets extra per-site labels, and we report results across that label budget.

What counts as winning. Beating the baselines means higher held-out-intersection AP than both the homography arm and the zero-label HEAL arm, with the improvement concentrated on vehicle-occluded objects. The method should also match HEAL fine-tuned on a nontrivial per-site label budget.

Why not just…?

Why not just use DG-BEV's homography augmentation? This is the most direct competitor. A homography is exact for camera rotation, but under height change it smears 3D objects and never hides one object behind another. Our ablation compares the two warps under identical pose distributions. If homographies match depth warps on the vehicle-occluded subset, our occlusion argument is wrong, and the simpler method should be preferred.

Why not just train or fine-tune a new encoder per roadside unit, as in HEAL? This is a real threat. If labeling a few hundred frames per intersection is cheap, per-site alignment may simply win. We settle it with a label-budget curve on held-out intersections. The claim survives only if zero-shot depth-warp training matches HEAL at a meaningful per-site budget, or if the two combine to beat HEAL alone.

Why not just put LiDAR on the roadside units? LiDAR fusion is clearly stronger: DAIR-V2X reports 41.90 versus 26.56 AP. Our question concerns camera-only deployments and LiDAR-failure fallback, the regime SiMO shows to be fragile. We use LiDAR only during training, as 3D-POP uses motion capture, and we report the remaining gap to LiDAR honestly.

Why not just use standard 2D augmentation? DG-BEV's 0.040 mAP baseline under rig change already shows that cropping and flipping do not simulate mounting changes. Our naive baseline re-checks this on cooperative data.

Proposed Experiments

Main split. On DAIR-V2X-C we hold out a subset of the 28 intersections, chosen to differ in roadside mount geometry, and train on the rest. We report cooperative camera-only 3D AP using the dataset's protocol.

Decisive check. The decisive check splits ground-truth objects by whether the vehicle's own LiDAR sees them. The proposal predicts that the advantage of visibility-corrected depth warping over homography warping is largest on vehicle-occluded objects. It also predicts that removing the visibility relabeling erases most of that advantage. If the gain is uniform across visibility classes, the mechanism is generic geometric robustness, not occlusion reasoning, and we would report that.

Simulation check. On OPV2V-H we use CARLA's exact depth to remove depth-completion error. This separates failures of the idea from failures of the depth estimates.

Tracking. We also run a simple tracker on the cooperative detections over DAIR-V2X sequences. We measure track breaks on objects that pass behind occluders, to see whether more consistent cooperative detections reduce dropouts.

Fallback. Finally, we test the trained camera encoder inside a SiMO-style fallback with LiDAR fully removed. This checks that better viewpoint robustness carries over to the camera-only regime where SiMO notes its ceiling.