Anchor Distributions, Not Anchor Sensors: Plug-In Sensor Adapters That Close the Modality Gap

Modular perception systems promise that you can add, remove, or replace a sensor without rebuilding the network. Current designs keep that promise only partly, because they keep one privileged camera stream as the anchor. CMNeXt routes depth, LiDAR and event data through a hub whose routing is driven by the RGB branch. ModTr translates thermal images into the input space of a frozen RGB detector. When the RGB camera goes dark or breaks, the parts that were supposed to be interchangeable no longer have anything to attach to. Separately, research on the "modality gap" shows that features from different modalities, even after joint training, sit in separate and linearly separable regions of the shared space. That second finding suggests why the anchor is hard to remove. Auxiliary-sensor features are only usable through RGB because they never landed in the same region as RGB features in the first place.

We propose to replace the anchor sensor with an anchor distribution. A frozen fusion-and-task head is paired with a stored target feature distribution, which we call the interface contract. Every sensor adapter, including the RGB one, is trained to emit features that match this contract. Matching is enforced class by class across the whole distribution, and also pixel by pixel against any sensor already integrated, rather than only through the task loss. If the gap is closed at the plug interface, the head should run on any subset of sensors, including subsets with no camera. A new sensor could then be added later by training only its own adapter. The proposal also tests a disputed question directly on physical sensor corruptions: does closing the gap cost the noise robustness that a gap is claimed to provide?

Core Claim

The claim is that anchor dependence in modular fusion is a geometric problem located at the adapter output, not a routing problem. A per-sensor adapter trained with three terms should do two things. The first term is the task loss. The second is a class-conditional distribution-matching term against the stored contract. The third is a pairwise term against co-registered features from an already-integrated sensor. Such an adapter should beat both task-loss-only adapters and RGB-anchored hubs on sensor subsets that lack RGB. It should also match jointly trained systems when sensors are added one at a time with everything else frozen.

For a new sensor $k$ with adapter $f_k$, the training objective is

$$ \mathcal{L}_k = \mathcal{L}_{\text{task}}\big(h(f_k(x_k))\big) + \alpha \sum_{c} \mathrm{MMD}^2\big(P_k^{c}, Q^{c}\big) + \beta\, \mathbb{E}\,\big\| f_k(x_k) - f_j(x_j) \big\|^2 . $$

Each symbol is defined as follows. $h$ is the frozen fusion-and-task head. $P_k^{c}$ is the distribution of adapter-$k$ pixel features whose label is class $c$. $Q^{c}$ is the stored contract distribution for class $c$, kept as a fixed bank of feature samples. The last expectation runs over spatially co-registered pixels of sensor $k$ and an already-integrated sensor $j$. $\alpha$ and $\beta$ are weights. Only $f_k$ receives gradients.

The three terms play distinct roles. The MMD term (maximum mean discrepancy, a kernel distance between two sample sets) comes from Learning Robust Visual-Semantic Embeddings. That work showed that matching pairs alone leaves global distributions apart, and used MMD to pull image and attribute distributions together for zero-shot classification. The pairwise Euclidean term follows the Align-True-Pairs idea of Closing the Modality Gap Aligns Group-Wise Semantics, which replaced cosine alignment with exact distance to collapse "semantic stripes."

We make the distribution term class-conditional for a specific reason. That same paper reports weaker gains on imbalanced data such as CIFAR10, and driving scenes are heavily imbalanced across classes.

Where This Sits

Three modular systems motivate the problem.

Delivering Arbitrary-Modal Semantic Segmentation introduces CMNeXt, an RGB branch plus a shared hub that scores and mixes auxiliary sensors at about 0.01M parameters per sensor. Its authors state that removing the RGB dependency is open. Our design does not route through RGB at all.

OmniSegmentor uses modality-specific stems and pairs RGB with one random auxiliary sensor per pre-training step. This aligns each sensor to RGB but never aligns auxiliary sensors to each other or constrains their global distribution. Our contract makes all sensors target the same distribution, so alignment between any two sensors follows from each matching the contract.

Modality Translation for Object Detection Adaptation Without Forgetting Prior Knowledge supplies the "adapt the sensor to a frozen model" method. Its adapter, ModTr, is trained purely by task loss, and it degrades on sparse inputs such as edge maps. We keep its frozen-head discipline but add explicit distribution constraints. We test whether those constraints help exactly where task loss alone is under-determined.

Two papers constrain the design. Is the Modality Gap a Bug or a Feature? A Robustness Perspective argues that a gap orthogonal to the modality subspaces cushions small perturbations. It also offers a training-free mean shift that closes the gap. If that argument transfers to physical sensors, fully closing the gap would trade away robustness. We therefore treat gap magnitude as a controlled variable rather than driving it to zero by assumption. Towards understanding the modality gap in CLIP shows that contrastive training settles into gapped local minima and that a linear probe separates image from text embeddings perfectly. We borrow that probe as a gap diagnostic.

Tuning LayerNorm in Attention frames cross-modal adaptation as distribution shift fixable by normalization parameters alone. This is the cheapest rival explanation of our effect, and we test it head-on.

Temporal misalignment between sensors, addressed by AsyncBEV, is deliberately out of scope. We use synchronized data so that spatial-temporal error does not confound the latent gap we are measuring.

Research Program

The first stage builds the contract. We take a pretrained segmentation encoder of the same scale as OmniSegmentor's and train an RGB+depth system on DELIVER. We then freeze the fusion-and-task head and store a class-conditional bank of a few thousand fused pixel features per class as the contract. After this point the RGB encoder is just another adapter. Fusion over whichever sensors are present is a mean of the contract-space features, weighted by a small per-pixel gate computed symmetrically from all present sensors rather than from RGB queries.

The second stage adds sensors one at a time under the objective above. Event and LiDAR adapters are trained one at a time, using pairwise partners drawn from any already-integrated sensor. Nothing else changes during this stage.

The third stage is the gap analysis. For each adapter we measure three quantities: linear-probe separability of sensor identity, centroid distance to the contract, and class-wise V-Measure of the pooled features. We then reintroduce a controlled offset of magnitude $\lambda$ along a direction orthogonal to the contract's principal subspace (the top principal components of the contract bank). Sweeping $\lambda$ maps clean accuracy against corrupted accuracy. This turns the bug-or-feature dispute into a measured curve on real failure modes rather than synthetic Gaussian noise.

Baselines

The naive reference points are two. The first is an RGB-only model. The second is the frozen RGB+depth head with missing sensors zero-filled. Together they show how much any method gains from extra sensors at all, and how badly the system fails when the anchor disappears.

The strongest existing competitor on DELIVER is OmniSegmentor. It reports 68.0% mean intersection-over-union (mIoU) on the four-modality RGB-Depth-Event-LiDAR task, per OmniSegmentor. It is the strongest because it is both modular and pre-trained at scale. CMNeXt is the second competitor, because it is the system explicitly designed for arbitrary sensor subsets. The file reports only relative figures for it: over 22% above mono-modal baselines under RGB under-exposure, and monotonic gains as sensors are added. For both we will retrain from released weights where available, at matched encoder size, on the same splits and with the same epoch and tuning budgets. Results will be reported on every one of the 15 non-empty sensor subsets, not only the full set.

Beating these competitors means two things. On the full four-sensor set we must stay within about one mIoU point of OmniSegmentor. On the seven subsets without RGB we must clearly exceed both systems, averaged over three seeds.

For the add-later protocol, the reference is the same architecture trained jointly on all four sensors. Matching it within noise counts as success.

To transfer the method to detection, we compare on LLVIP against ModTr with the same frozen FCOS and RetinaNet detectors. The file gives only a relative figure for ModTr: up to 29.0 AP over HalluciDet. No absolute LLVIP AP was found on Lacuna, so we will rerun ModTr ourselves. No reported FLIR ALIGNED number was found on Lacuna for ModTr either.

The key ablation removes the distribution and pairwise terms, leaving task-loss-only adapters, which is ModTr's training applied to segmentation. A second ablation keeps only per-sensor re-centering, the first-moment part of the gap. Together these isolate whether full distribution matching is the active ingredient.

Why not just…?

Why not just train with random modality dropout, including dropping RGB? This is the most serious threat to the anchor-free claim. A jointly trained network that sometimes sees no camera may learn to cope without one. We include it as a baseline with RGB-drop probability tuned on validation. Dropout cannot, however, deliver the add-later property: every new sensor requires retraining everything. Our method also predicts a specific advantage that dropout lacks, namely smaller gains on sparse sensors. If dropout matches us on RGB-free subsets, the anchor-removal half of the claim reduces to that known trick. The remaining contribution would then be incremental integration, and we will say so.

Why not just re-normalize each sensor's features, or tune only normalization layers? Tuning LayerNorm in Attention shows that tuning normalization parameters alone (0.004% of weights) outperformed LoRA on vision-language transfer. The orthogonal mean shift of Is the Modality Gap a Bug or a Feature? closes the gap with no training. If the sensor gap is mostly a mean offset, either would do. Neither paper tested non-linguistic physical sensors, and LayerNorm tuning is reported to need an already partly aligned connector. The re-centering ablation and a LayerNorm-only variant settle this. If they recover most of our gain on RGB-free subsets, we report that the gap at sensor interfaces is first-order, and we recommend the cheap fix.

Why not just use ModTr-style task-loss adapters into a frozen head? They already preserve the base model exactly. But task loss only constrains features along directions the head reads, which leaves the adapter's output distribution unconstrained elsewhere. ModTr reports steep failure on sparse edge inputs. Our ablation measures whether the distribution term narrows that failure on LiDAR and event data, which are sparse in DELIVER.

Why not just retrain everything jointly? Joint retraining is the add-later reference. The ModTr paper reports that joint multi-modal training degrades original RGB COCO accuracy, and it must be redone for every hardware change. If joint training is no better than incremental adapters, there is no reason to pay that cost.

Decisive Checks

The proposal stands or falls on three measurements on DELIVER.

The first is mean mIoU over the seven RGB-free subsets. Contract adapters must beat CMNeXt, OmniSegmentor, task-loss-only adapters, and re-centering-only adapters. Comparison against tuned modality dropout determines how much of the gain is specific to our method.

The second is the add-later gap. Adapters integrated one at a time must land within noise of joint training on the full set and on every subset.

The third is the $\lambda$ sweep on DELIVER's corruption modes (LiDAR jitter, motion blur, over-exposure, under-exposure) and its weather conditions. The robustness argument predicts that a small nonzero orthogonal offset beats both $\lambda = 0$ and the natural gap on corrupted data at little clean cost. A flat or monotone curve would indicate that the robustness argument does not carry over to physical sensor failure. Alongside these, the linear-probe separability of sensor identity should fall toward chance on the semantic subspace for our adapters but not for the baselines. If it does not fall, any accuracy gain cannot be credited to closing the gap.