Self-driving perception systems combine surround cameras with LiDAR, a laser scanner that returns sparse but precise 3D points. At night the camera half of these systems degrades badly: on nuScenes, vision-based detection loses 55–58% of its mean average precision after dark, against 36% for LiDAR (nuScenes). The usual fix for a lighting gap is unpaired image-to-image translation. This family of methods, CycleGAN being the best known, learns to turn night images into day-looking images without matched day/night pairs of the same scene. Applied one camera at a time, it produces geometry that disagrees across views (Multi-View Image-to-Image Translation Supervised by 3D Pose), and multi-view 3D detection is very sensitive to such inconsistencies (UDGA). Round-trip cycle consistency also assumes both domains carry the same information, which is false when night images have lost most of their texture.
This note proposes a different constraint. Day and night recordings are never paired with each other, but inside every night recording the camera and the LiDAR are paired, and LiDAR geometry barely depends on ambient light. We therefore train one small translator, shared by all cameras, that converts night camera features into day-like camera features before they are lifted into a bird's-eye-view (BEV) map, a top-down metric grid on which all cameras and the LiDAR are fused. The translator is constrained in that shared map by the simultaneous LiDAR scan rather than by a cycle back to the night image. In short, the concurrent LiDAR stands in for the missing paired day image, and the shared BEV grid makes one translator handle every camera consistently.
Core Claim
Our claim is that a translator shared across views, applied to camera features before depth estimation and lifting, and trained without day/night pairs, will improve night-time camera and fused detection on nuScenes more than three alternatives: feature-level adversarial adaptation, per-view pixel translation, and direct LiDAR-supervised self-training of the camera branch. We use three training signals, all computed after lifting into BEV. The first makes the distribution of translated features match day camera BEV features. The second requires the occupancy implied by the translated camera features to agree with occupancy from the concurrent LiDAR sweep. The third requires overlapping cameras to agree on depth for the same world location. We expect the gains to concentrate where the camera matters most: long range, where LiDAR returns are sparse, and LiDAR dropout, where the camera is the only sensor left.
Relation to Prior Work
Three groups of papers shape the design.
The first motivates it. Diamant et al. couple one CycleGAN per camera through a loss from a frozen 3D pose estimator, which prevents "pose collapse" on CMU-Panoptic. It needs a separate generator and discriminator per view and applies its 3D constraint only after images are generated. We use one translator conditioned on each pixel's viewing ray and impose the constraint in a shared metric grid. BEVFusion (Liu et al.) and BEVFusion (Liang et al.) establish BEV as the right place for early camera–LiDAR fusion. Liang et al. also show that an independent camera stream rescues detection when LiDAR fails, which is the regime our translator targets. Both models are trained and evaluated only on in-domain paired data.
The second group constrains the design. ECFusion shows two failure modes in BEV fusion: depth errors displace camera features into "ghost" locations, and sparse LiDAR can dissolve strong camera evidence. Both get worse when camera features are corrupted by darkness. We therefore translate before lifting, so the depth network sees repaired features, instead of translating finished BEV maps in which the depth errors are already baked in. The claim that cycle consistency fails under information asymmetry, from the unpaired-translation literature summarized in the file, rules out a round-trip loss. Countering Multi-modal Representation Collapse warns that fused representations can collapse onto the dominant modality. That warning shapes both the LiDAR anchor and the control against it: the anchor constrains only occupancy, never the semantic channels, so that the translator cannot simply copy LiDAR.
The third group supplies the competing adaptation strategies. DA-BEV aligns object queries adversarially across domains and self-trains, lifting camera-only day-to-night mAP from 15.92% to 20.27%. It never repairs low-level features and uses no LiDAR. xMUDA has 2D and 3D networks imitate each other's output predictions on target data, taking 2D day-to-night segmentation from 42.2 to 47.1 mIoU. It uses the same "the other sensor is a teacher" intuition as we do, but at the output layer rather than inside the feature hierarchy. UDGA supplies the overlap-consistency idea, using agreement between adjacent cameras as self-supervision. We reuse it as a regularizer rather than as a response to shifts in camera mounting.
Research Program
We start from a BEVFusion-style model (Liu et al.) trained on nuScenes daytime scenes and keep its camera backbone, depth network, LiDAR branch and heads frozen. Freezing isolates what the translator contributes and makes it a plug-in for existing deployed models. The translator $T$ is a lightweight residual convolutional network inserted after the image backbone. It is conditioned on a per-pixel embedding of the viewing ray (Plücker coordinates, following Campbell et al.), so one set of weights can serve all six cameras and rigs with different mounting. The translated features go through the frozen depth-and-lift step to produce a camera BEV map $B_c$.
Training uses unlabeled night scenes, each with its concurrent LiDAR sweep, together with unpaired day scenes. The objective is
$$\mathcal{L}(T) = \mathcal{L}_{\text{dist}} + \lambda_{\text{occ}}\,\mathcal{L}_{\text{occ}} + \lambda_{\text{ov}}\,\mathcal{L}_{\text{ov}} + \lambda_{\text{id}}\,\mathcal{L}_{\text{id}}.$$
Each term does a specific job:
- $\mathcal{L}_{\text{dist}}$ is a one-directional adversarial loss. A discriminator on BEV patches tries to tell translated night $B_c$ from day $B_c$.
- $\mathcal{L}_{\text{occ}}$ is a binary cross-entropy between the occupancy predicted from the translated $B_c$ and the occupancy obtained by voxelizing the concurrent LiDAR sweep. The occupancy head is a small head trained once on day data and then frozen. This term is the lighting-invariant anchor that replaces cycle consistency.
- $\mathcal{L}_{\text{ov}}$ penalizes disagreement between the depths two overlapping cameras assign to the same world point.
- $\mathcal{L}_{\text{id}}$ asks $T$ to leave day features unchanged, so the translator is a no-op in the domain where the model already works.
Sparse LiDAR can only supervise occupancy where it has returns. We therefore restrict $\mathcal{L}_{\text{occ}}$ to BEV cells that contain LiDAR points or lie along a ray the LiDAR has confirmed empty. Cells with no LiDAR information are left free rather than pushed toward "empty."
The analysis side asks two questions: where the translator changes the depth distribution of the camera branch, and whether it reduces ECFusion-style ghost features. For the second, we measure how much camera BEV activation falls in cells that LiDAR marks as free space.
Baselines
The naive reference point is the day-trained BEVFusion model evaluated unchanged on night scenes, run both with LiDAR and camera-only. It sets the size of the gap. No reported result found on Lacuna for BEVFusion or ECFusion restricted to nuScenes night scenes, so we reproduce these numbers ourselves with public code. For orientation, the file reports ECFusion at 70.7% mAP / 73.4% NDS and Liang et al.'s BEVFusion at 69.2% mAP on the full nuScenes set.
Feature-level adversarial adaptation. This is DA-BEV with its query adversarial learning and self-training. It is the strongest published competitor on this exact shift, reporting camera-only day-to-night mAP of 20.27% (from 15.92%). We run it twice: camera-only, to compare against its reported number, and grafted onto the same fusion model, using the same unlabeled night data and the same number of GPU-hours and hyperparameter trials as ours. Beating it means a higher camera-only night mAP than a faithful reproduction of 20.27%, and a higher fused night mAP/NDS than the grafted version, averaged over three seeds with non-overlapping spread.
Output-level cross-modal imitation. This is xMUDA-style imitation adapted to BEV detection and segmentation heads. It is the closest competitor that also uses LiDAR as a teacher for the camera. We apply it to the same frozen backbone with its heads unfrozen and give it equal tuning.
Per-view pixel translation. A CycleGAN is trained on night-to-day nuScenes images, and every camera is passed through it before the frozen model, as the "preprocessing" option. A second version adds the Diamant et al. coupling, with a detector-consistency loss in place of their pose loss, to test whether coupling alone closes the gap.
LiDAR-supervised self-training. This is the most dangerous simple baseline. It fine-tunes the camera depth network on night data with projected LiDAR depth and fine-tunes the camera branch on pseudo-boxes from the LiDAR branch. It uses the same signal we do, without a translator. It gets the same trainable parameter count as our translator, or alternatively the full camera branch, and the same compute budget.
Ablations that isolate our key ingredient:
- drop $\mathcal{L}_{\text{occ}}$, leaving a distribution-only translator;
- replace $\mathcal{L}_{\text{occ}}$ with a feature-cycle loss, to test the asymmetry argument;
- move $T$ after lifting, translating in BEV only;
- use per-view translators instead of the shared, ray-conditioned one;
- drop $\mathcal{L}_{\text{ov}}$.
The proposal's central claim survives only if removing the LiDAR anchor, or swapping it for a cycle loss, clearly hurts night performance.
Why not just…?
Why not just run separate camera and LiDAR detectors and merge their boxes? Late fusion cannot recover an object that neither detector proposes, and the night camera detector will miss many. BEVFusion (Liu et al.) reports that fusing in BEV beats point-level combination, with +1.3% mAP/NDS on detection and +13.6% mIoU on BEV map segmentation. We do not re-argue that design choice. Our translator operates inside an early-fusion model, and a late-fusion ensemble is included only as a sanity point in the LiDAR-dropout table.
Why not just translate each camera image with a 2D CycleGAN first? Independent per-view generators produce geometry that conflicts across cameras (Diamant et al.). Multi-view lifting is fragile enough that small geometric perturbations cost up to 54% of detection score (UDGA), and texture hallucinated by a pixel generator feeds straight into depth estimation. We test this directly, including the coupled variant. If pixel translation matches us, the conclusion is that the LiDAR anchor is unnecessary, and we would report that.
Why not just align features adversarially or have the modalities imitate each other, as in DA-BEV or xMUDA? Both leave large gaps. DA-BEV ends at 20.27% night mAP, and xMUDA's imitation adds 4.9 mIoU. Neither changes the low-level features the depth network consumes, which is where ECFusion locates the ghost artifacts. Still, a grafted DA-BEV on a fusion model is untested in the file and could be competitive. The matched-budget comparison settles this, and a combined DA-BEV-plus-translator run shows whether the two are complementary.
Why not just fine-tune the camera branch at night with LiDAR depth and LiDAR pseudo-labels? This is a real threat, because it uses the same free night supervision with less machinery. Our argument is conditional. Direct fine-tuning changes the deployed model and can hurt daytime accuracy, while our translator is identity on day data by construction. And the distribution term in our loss supplies day-like semantic texture, which LiDAR cannot provide. If this baseline equals us on night accuracy and does not lose daytime accuracy, the translation component is not justified. In that case we would reframe the contribution as the occupancy-anchor recipe itself, applied to self-training.
Decisive Experiments
All main experiments use nuScenes, splitting scenes by the day/night and Boston/Singapore metadata used by DA-BEV and xMUDA. We report detection mAP and NDS, and BEV map segmentation mIoU, on night validation scenes under four conditions: full sensors, camera-only, simulated LiDAR dropout following the failure protocol of BEVFusion (Liang et al.), and range bins beyond 30 m. We also report day validation accuracy to catch regressions.
Four checks decide whether the claim holds:
- Anchor ablation. Camera-only night mAP must drop measurably when $\mathcal{L}_{\text{occ}}$ is removed or replaced by a cycle loss. If it does not, the core claim fails.
- Self-training comparison. Under LiDAR dropout, the translator must beat LiDAR-supervised self-training at equal compute without lowering day accuracy.
- Shared versus per-view. The shared ray-conditioned translator must match or beat per-view translators while also transferring to an unseen camera. We test this by training on five cameras and evaluating on the sixth.
- Ghost features. The fraction of camera BEV activation falling in LiDAR-confirmed free space should fall at night, which would tie the accuracy gain to the ECFusion mechanism rather than to generic regularization.
As a secondary test of transfer to a different shift, we rerun the pipeline on nuScenes rain scenes. In rain, LiDAR also degrades, so the anchor is weaker. That run shows how much the method depends on the assumption that LiDAR is unaffected.
References
- nuScenes: A multimodal dataset for autonomous driving
- Multi-View Image-to-Image Translation Supervised by 3D Pose
- Unified Domain Generalization and Adaptation for Multi-View 3D Object Detection
- BEVFusion: Multi-Task Multi-Sensor Fusion with Unified Bird's-Eye View Representation
- BEVFusion: A Simple and Robust LiDAR-Camera Fusion Framework
- Eliminating Cross-modal Conflicts in BEV Space for LiDAR-Camera 3D Object Detection
- Countering Multi-modal Representation Collapse through Rank-targeted Fusion
- DA-BEV: Unsupervised Domain Adaptation for Bird's Eye View Perception
- xMUDA: Cross-Modal Unsupervised Domain Adaptation for 3D Semantic Segmentation
- PluckeRF: A Line-based 3D Representation for Few-view Reconstruction