Gap-Gated Bidirectional Camera–LiDAR Fusion for Sim-to-Real Driving Perception

Self-driving perception models are cheap to train in simulators like CARLA, where every pixel and point comes with perfect labels. They collapse when they reach real roads. The obvious fix is to fuse camera and LiDAR (the laser range sensor) so that each modality can correct the other, and to align simulated and real features with adversarial discriminators. A discriminator here is a small classifier that tries to tell simulated features from real ones, while the main network learns to fool it. This note argues that the obvious fix has a hidden failure. The simulator-to-reality gap is lopsided: it is large for camera appearance (textures, lighting, lens effects) and smaller for LiDAR geometry. Which modality is closer to reality also changes with representation level and over time. Symmetric two-way attention between modalities ignores this. It lets the camera branch write its large appearance gap into LiDAR features, which undoes the one modality that transfers reasonably well.

The proposal is to measure the sim-to-real gap separately for each modality and at each level of the network: per-frame image features, the top-down bird's-eye-view map, and the change in that map over time. Those measurements then set how strongly each direction of cross-modal attention is allowed to flow at that level. At every level, the better-aligned modality may write into the worse-aligned one, but not the reverse. Put plainly, invariance to the domain should be passed from the modality that already has it to the one that lacks it, and the flow should be decided by measurement rather than fixed by architecture.

Core Claim

Our claim has two parts. The first is that unconstrained bidirectional cross-modal attention, trained with standard multi-level adversarial alignment on CARLA and tested on nuScenes, makes the LiDAR pathway worse than a LiDAR-only model aligned the same way. We call this contamination. The second is that gating each attention direction by a per-level, per-modality estimate of the domain gap removes the contamination and still keeps the camera's benefit for object identity. The research file already names "asymmetric modality capacities" as an open problem for bidirectional attention. No paper in it studies that asymmetry when the asymmetry is itself caused by domain shift and differs across levels.

The closest work fixes the direction of cross-modal flow in advance or keeps the modalities apart:

  • DualCross distills a LiDAR teacher into a camera student in one direction only. It aligns at image and BEV levels and is tested only on real-to-real shifts.
  • xMUDA makes 2D and 3D networks mimic each other's outputs symmetrically, through a KL-divergence loss at the prediction layer, and has no intermediate attention.
  • UPIDet propagates features both pixel-to-point and point-to-pixel, but only in-domain.
  • BCAT applies bidirectional cross-attention between source and target domains within one image modality.

Our proposal differs from all four in the same way. The direction of flow between modalities becomes a quantity that the domain gap controls, and it is set separately at each level of the hierarchy.

Research Program

The base architecture is a BEVFormer-style model (BEVFormer) extended with a LiDAR pillar branch. BEVFormer uses grid-shaped bird's-eye-view queries, gathers camera features through spatial cross-attention, and carries history through temporal self-attention that warps the previous BEV map by ego-motion. We add cross-modal attention in both directions at two places:

  • Image level: LiDAR points projected into image space attend to camera patches, and the reverse, in the spirit of UPIDet.
  • BEV level: camera-BEV and LiDAR-BEV maps attend to each other.

We also add a third, temporal level. Each modality's BEV is warped into the current frame and compared with the current BEV, and the residual is aligned between simulation and reality. This residual is where unrealistic simulated vehicle motion and over-smooth ego-motion should show up. The file flags exactly this temporal discrepancy as unaddressed by static image alignment.

Each level $l$ and modality $m \in \{c, L\}$ (camera, LiDAR) has a discriminator trained with the Pearson $\chi^2$ $f$-divergence objective. We use that objective because Towards Optimal Strategies found latent-space alignment with it more effective than image-space style transfer. From each discriminator we read a running, held-out divergence estimate $\hat g_l^m$, which is the estimated sim-to-real gap of modality $m$ at level $l$. The strength of the attention path that lets modality $a$ write into modality $b$ at level $l$ is

$$\alpha_l^{a\to b} = \sigma\!\left(\beta\,(\hat g_l^{b} - \hat g_l^{a})\right),$$

where $\sigma$ is the logistic function and $\beta$ is a temperature. When $a$ is better aligned than $b$, so that $\hat g_l^a < \hat g_l^b$, the path is mostly open. When $a$ is the worse-aligned modality, the path is mostly closed. The gates scale the residual added by each cross-attention path. They are recomputed during training, so the flow can reverse as alignment improves.

Two further pieces keep the comparison with prior work fair. We use the spatial prior from Towards Optimal Strategies, which matches where vehicles appear in simulation to real-world statistics, because that paper shows the label-distribution gap sets a floor on error that alignment alone cannot remove. We also use target pseudo-labels that must agree across frames after ego-motion warping.

The analysis component matters as much as the model. For each level we will log the gap estimates over training. We will also measure contamination directly, as the target-domain accuracy of a LiDAR-only probe head attached to the LiDAR features of the fused model, compared against an aligned LiDAR-only model.

Baselines

Unadapted reference. The do-nothing reference is the same fused architecture trained on CARLA and evaluated on nuScenes with no alignment. Next to it we report the unadapted single-modality numbers from Towards Optimal Strategies: 9.76 IoU for camera BEV vehicle segmentation, as stated in the file.

Spatial prior with $f$-divergence alignment. This is the strongest published sim-to-real method on this exact task. It reports 17.84 IoU for camera and 17.20 IoU for LiDAR BEV vehicle segmentation on CARLA→nuScenes. The file also gives 14.41 IoU for LiDAR, once as the unadapted figure and once as a "standard adversarial baseline", and we will note this ambiguity rather than resolve it. The method is single-frame and single-modality, so we rerun it with its spatial prior and pseudo-labeling on our regenerated sequential CARLA data. We use the same nuScenes target split and backbone sizes, and the same hyperparameter search budget as our method. Beating it means exceeding the better of the rerun camera and LiDAR models by a margin larger than the seed-to-seed spread over three seeds. This bar matters because a fused model that cannot beat the better single modality has not justified fusion.

xMUDA-style late fusion. This is the strongest cross-modal adaptation alternative. It keeps two separate branches, applies output-level mimicry in both directions with pseudo-labels, and adds the same spatial prior. It isolates whether attention between intermediate features is needed at all. No reported result was found on Lacuna for xMUDA on CARLA→nuScenes; its Day→Night 2D result of 47.1 mIoU is on a different task and is not directly comparable.

DualCross-style distillation. A one-way LiDAR→camera variant with image-level and BEV-level discriminators, which tests whether a fixed direction already captures the benefit.

Ablations. Three ablations isolate the key ingredient:

  • ungated symmetric bidirectional attention under the same discriminators;
  • learned gates that are free parameters and receive no gap signal;
  • gap gating with the temporal level removed.

Why not just…?

Why not just use LiDAR alone with the spatial prior, since geometry transfers better? This is the most serious threat. If the camera adds nothing once alignment is done, our method reduces to an expensive LiDAR model. The file only weakly suggests a camera benefit: Towards Optimal Strategies lists pedestrian-versus-vehicle confusions as remaining errors, and appearance should help resolve object identity. Our main comparison is against the better single modality, and a per-class breakdown will show whether any camera benefit is concentrated on identity-ambiguous classes. If fusion does not beat LiDAR-only, we will report that as the finding.

Why not just let attention learn the gates, since training can discover which modality to trust? Training supervises source labels only. On simulated data the camera is clean and highly informative, so ungated or freely learned gates should learn to trust the camera exactly where it will mislead at test time. The learned-gate ablation tests this directly. If it matches gap gating, the gap signal is unnecessary and the claim fails.

Why not just fuse at the output with mutual mimicry, as xMUDA does? Output-level mimicry cannot contaminate intermediate features, so it is immune to the failure we describe. It also cannot let LiDAR geometry shape camera features before the BEV projection. The xMUDA-style baseline settles which effect dominates.

Why not close the gap in pixel space with style transfer or randomization? Towards Optimal Strategies reports that latent-space alignment beats raw image-space style transfer for this task, so we do not treat that route as the primary competitor.

Decisive Experiments

Data. The main benchmark is CARLA→nuScenes BEV vehicle segmentation measured by IoU. We follow the source setup of Towards Optimal Strategies (CARLA, about 4,000 episodes), but record consecutive frames so that the temporal level is meaningful. nuScenes supplies consecutive sweeps and ego poses. As a controlled check, we use SHIFT, a synthetic dataset with gradual weather and lighting change, to vary appearance gap and dynamics gap independently. SHIFT reports a semantic segmentation drop from 83.6 to 42.8 mIoU when moving from day to night. We also run the nuScenes Day→Night split, where DualCross reports 17.0 IoU for vehicle segmentation. This tests whether gating helps when the gap is purely appearance.

First check: contamination exists. Under ungated bidirectional attention, the LiDAR probe head should fall below an aligned LiDAR-only model on nuScenes. If it does not, the motivating premise is false and the gating story collapses.

Second check: gating removes contamination and fusion still pays. Gap-gated fusion should beat both the ungated and learned-gate variants, and should exceed the best single-modality rerun. The logged gates should show camera→LiDAR flow suppressed at the image level more than at the BEV level.

Third check: the gate tracks the actual gap. On SHIFT, as night-time darkening increases the camera gap, the camera→LiDAR gate should close monotonically, and target accuracy should degrade more gracefully than with ungated fusion.

Fourth check: the temporal level pays for itself. Removing the temporal-residual level should hurt most on nuScenes velocity-sensitive and occluded objects, where BEVFormer's recurrence matters. It should hurt little on static frames.