Self-driving cars carry both cameras and LiDAR, a laser scanner that returns a 3D point cloud. When a perception model trained on daytime data is deployed at night, the camera degrades far more than the LiDAR. The nuScenes dataset paper reports that vision-based detection loses 55–58% of its accuracy at night while LiDAR loses 36% (nuScenes). Current methods for adapting such two-sensor systems to a new domain without target labels retrain both networks end to end and let each sensor supervise the other. This lets a blinded camera teach the LiDAR network wrong answers. We propose a different design. Both backbones stay frozen: a large pre-trained image model such as DINOv2 and a source-trained 3D network. The only thing that learns is a small module in which each LiDAR point acts as a query. It looks up the image model's features across several layers, in a small window around where the point lands in the image, and it then passes a learned per-point gate that can shut the image contribution off entirely.
The idea fits in two sentences. Because neither backbone is ever updated, a failing camera cannot damage the LiDAR representation; the worst it can do is add noise through the gate, and closing the gate recovers the LiDAR-only prediction exactly. Letting each point choose which depth of the image model to read lets the system use shallow, texture-level layers when deep semantic layers break down at night, and the reverse when they do not. All of this costs well under 1% of the parameters and none of the backpropagation memory of full retraining.
Core Claim
A point-conditioned, multi-layer, sparsely gated cross-attention module over frozen backbones can match the domain-adaptive 3D segmentation accuracy of full-retraining methods on nuScenes Day→Night and A2D2→SemanticKITTI. Unlike them, it satisfies a hard floor under camera failure: fused accuracy never falls below the LiDAR-only prediction. The claim has three parts, and each is individually falsifiable. First, the frozen-backbone module reaches the adaptation accuracy of end-to-end methods. Second, the gate is what provides the robustness floor. Third, attention over layers, rather than reading only the final layer at the projected pixel, carries measurable signal under the night shift.
Where This Sits in Prior Work
Two papers motivate the problem. xMUDA trains a 2D and a 3D network jointly, with each network's extra prediction head imitating the other's output on unlabeled target data. It lifts the 2D stream on Day→Night from 42.2 to 47.1 mIoU with pseudo-labels, and the 3D stream on A2D2→SemanticKITTI from 36.6 to 48.5 mIoU. Its weakness is exactly the imitation: a wrong camera pulls the LiDAR network toward wrong predictions. Fusion-then-Distillation (FtD++) names this "negative mimicry." It fixes it by fusing the two streams through a memory-bank attention module and distilling from the fused result, and it reports +11.8% mIoU over source-only and +9.4% over xMUDA on Day→Night. Both methods still backpropagate through full 2D and 3D backbones. We keep their pseudo-label training recipe but remove the channel through which negative mimicry acts: shared, trainable backbone weights.
Three lines of work supply the mechanism. Attentive Multi-Layer Fusion (ALF) shows that a learned query attending over tokens from every layer of a frozen vision transformer beats last-layer linear probing by 5.54 points on average, and by up to 30 on specialized data. Its query, however, is one fixed task vector over pooled tokens. We make the query a per-point LiDAR feature and keep spatial tokens. ComBo fuses patch tokens from several frozen image models with a light transformer and shows that keeping spatial structure matters, with +22% over linear probing on structured VTAB-1k tasks. It never meets a second sensor or a domain shift. Gated cross-attention for drug–target interaction supplies the gate-plus-sparsemax design. Sparsemax is a softmax variant that can assign exactly zero weight, so it stops one input from dominating. We reuse that design to suppress a corrupted sensor rather than a dominant molecule.
Two papers constrain the design. Side-Tuning shows that a frozen base plus an additive side network rules out forgetting, but its blending weight is one fixed scalar; our gate varies per point and per scene. M2IST already puts cross-modal interaction adapters beside frozen vision and language encoders. It does supervised referring-expression grounding with no domain shift, and its adapters are tied to specific encoder layers. So "frozen backbones plus cross-modal side adapters" is not new by itself. What is new here is its use for unlabeled cross-sensor adaptation, together with the robustness floor. The closest possible competitor is the UniDSeg repository, described only as cross-domain 3D segmentation using visual foundation model priors. The file gives no detail on whether it freezes backbones or gates the camera, so we treat it as a required baseline rather than claim a difference we cannot verify.
Research Program
For each LiDAR point $p$, let $h_p$ be its feature from the frozen 3D network and $u_p$ its projection into the camera image. Let $z_{\ell,j}$ be the frozen image model's patch token at layer $\ell$ and patch $j$, restricted to a $3\times3$ patch window around $u_p$, taken from a chosen subset of layers. The module computes
$$a_p = \sum_{\ell,j} \operatorname{sparsemax}_{\ell,j}\!\Big(\tfrac{q(h_p)^\top k_\ell(z_{\ell,j})}{\sqrt d}\Big)\, v_\ell(z_{\ell,j}), \qquad f_p = h_p + g_p\, a_p,$$
The terms are defined as follows:
- $q$, $k_\ell$ and $v_\ell$ are small learned projections, with a separate key and value projection for each layer $\ell$.
- $d$ is the projection width.
- $g_p \in [0,1]$ is a gate computed from $h_p$, $a_p$ and two summary statistics of the attention: its maximum weight and the number of nonzero entries.
- $f_p$ is passed to a new segmentation head.
Two properties follow from this design. If $g_p=0$, then $f_p=h_p$ and the head sees exactly the LiDAR feature. We initialise that head from the source LiDAR head, so a closed gate reproduces the source 3D prediction. The cost is linear in the number of points, about a hundred keys per point, which avoids the full point-by-patch attention that the research file flags as intractable on edge hardware.
Training proceeds in three stages:
- Train the module and head on labeled source data.
- Train on unlabeled target data with the xMUDA/FtD++ style of confident pseudo-labels.
- Add a regulariser that pushes $g_p$ toward zero where the fused and LiDAR-only predictions disagree and the LiDAR-only prediction is the more confident. This teaches the gate to defer to the LiDAR stream when the camera is unreliable, without any corruption labels.
The per-layer key and value projections are the only part tied to a specific image backbone. This lets us run a portability study: we train with DINOv2, swap in CLIP or SAM features, and retrain only $k_\ell$ and $v_\ell$ for a few minutes. That tests the architecture-agnostic claim directly instead of asserting it.
Baselines
The naive reference point is the source-only model, meaning the frozen 3D network and its original head with no adaptation. It also serves as the floor our gated module must never fall below under camera failure. The second reference is a frozen-feature concatenation head. It samples the last-layer DINOv2 token at each point's projected pixel, concatenates it with $h_p$, and trains an MLP head with the identical pseudo-label schedule, the same number of trainable parameters (within 10%) and the same tuning budget of hyperparameter trials. This is the baseline the skeptic expects to be sufficient.
The strongest existing methods are FtD++, run from the official FtD-PlusPlus code, and xMUDA with pseudo-labels, run through the BFtD-xMUDA code. The published numbers are 47.1 mIoU for the 2D stream on Day→Night for xMUDA with pseudo-labels, 48.5 3D mIoU on A2D2→SemanticKITTI for xMUDA, and FtD++'s reported +9.4% over xMUDA on Day→Night. The file gives no absolute FtD++ mIoU, so we re-run it and report both our reproduction and the published relative gain. Absolute numbers for USA→Singapore and VirtualKITTI→SemanticKITTI were not found on Lacuna. FtD++ and xMUDA retrain their backbones, which is their advantage. We therefore also run them with a DINOv2-initialised 2D backbone, so the image model is not what separates us from them. UniDSeg is run from its repository under the same splits. Beating FtD++ means matching its 3D and fused mIoU within one point while training under 1% of its parameters. Exceeding it by more than one point on the camera-degraded evaluation described below would count as a clear win.
Three ablations isolate the key ingredients:
- Removing the gate, with $g_p = 1$ fixed, tests the robustness floor.
- Restricting attention to the final layer tests the depth-selection claim.
- Replacing the point-conditioned query with a single learned ALF-style query per layer tests whether point conditioning matters.
A fourth, training-free baseline follows PEA. It re-whitens and re-colours the frozen image tokens toward target statistics in forward passes only, and feeds them to the source-trained module.
Why not just…?
Why not just concatenate frozen image and LiDAR features and train a head? Because LiDAR points project to known pixels, this could genuinely capture most of the gain. This is the most serious threat. The evidence against it is indirect. ALF and ComBo show that last-layer, pooled features lose substantial accuracy to attentive multi-layer fusion on shifted or structured tasks, and a concatenation head has no mechanism to ignore a black camera. The concatenation baseline and the final-layer ablation settle it directly. If our module does not beat concatenation by at least one mIoU point on clean Day→Night, and does not hold the LiDAR-only floor where concatenation drops below it under blackout, the attention component is unjustified. In that case we would reduce the proposal to "concatenation plus gate."
Why not just use a scalar gate driven by image brightness or confidence? Night failure is spatially uneven. Headlights, street lamps and dark regions coexist in one frame, and a global scalar, like Side-Tuning's fixed blending weight, cannot express that. A per-point gate over the final-layer feature is still a strong cheap alternative, so we include it. We report fused mIoU separately for points in lit and unlit image regions, split by image intensity at the projected pixel. That shows whether the multi-layer attention matters beyond the gate.
Why not just retrain everything, as FtD++ does? FtD++ already attacks negative mimicry and may well stay more accurate on clean target data. Our argument is that it provides no floor, and that retraining costs memory and ties the adaptation to one pair of backbones; ComBo reports over 21 GB for fine-tuning four backbones against 3.2 GB for frozen probing. We will not claim higher clean accuracy unless we measure it. The comparison that decides the question is accuracy under camera degradation together with training memory and time.
Why not just align feature statistics without training, as PEA does? PEA assumes the shift is a change in mean and covariance of one stream. Night imagery is plausibly a larger, non-linear change, and PEA has no way to fall back on a second sensor. The PEA-style baseline measures how much of the gap is closed by statistics alone. If it closes most of it, the learned module should be combined with alignment rather than replace it.
Proposed Experiments
The main benchmarks are nuScenes Day→Night and USA→Singapore, A2D2→SemanticKITTI, and VirtualKITTI→SemanticKITTI. All use the xMUDA splits and report 2D, 3D and fused mIoU, trainable parameters, peak training memory and training time.
The decisive check is a camera-degradation sweep on the Day→Night target set. At test time we black out, blur or add noise to a growing fraction of camera frames, from 0 to 100%. We plot fused mIoU against the source LiDAR-only line for four systems: the gated module, the ungated module, concatenation and FtD++. The claim succeeds if the gated module stays at or above the LiDAR-only line at every level. It should do so while the ungated variant and FtD++ cross below that line, and while the gated module matches FtD++ on unperturbed data.
A second check logs where the attention weight goes across layers for day and night target points. If night points do not shift weight across layers relative to day points, the depth-selection motivation borrowed from ALF does not transfer to this setting. The portability study then reports how much of the DINOv2-trained module's gain survives a backbone swap when only the per-layer projections are retrained, compared with training from scratch on the new backbone.
References
- nuScenes: A multimodal dataset for autonomous driving
- xMUDA: Cross-Modal Unsupervised Domain Adaptation for 3D Semantic Segmentation
- Fusion-Then-Distillation: Toward Cross-Modal Positive Distillation for Domain Adaptive 3D Semantic Segmentation
- Attentive Multi-Layer Fusion for Vision Transformers
- Fantastic Features and Where to Find Them: A Probing Method to combine Features from Multiple Foundation Models
- An Interpretable Framework for Drug-Target Interaction with Gated Cross Attention
- Side-Tuning: A Baseline for Network Adaptation via Additive Side Networks
- M2IST: Multi-Modal Interactive Side-Tuning for Efficient Referring Expression Comprehension
- Architecture-Agnostic Test-Time Adaptation via Backprop-Free Embedding Alignment