Unimodal Heads as Test-Time Teachers Under Asymmetric Sensor Shift

Multimodal classifiers, such as an action recognizer that watches video and listens to audio, are often trained with an extra small classifier ("head") on each sensor's encoder. The extra heads keep the easy modality from drowning out the harder one during training. At deployment the heads are usually thrown away. Yet deployment is exactly where sensors fail unevenly: fog blurs the camera while the microphone is fine, or wind noise ruins the audio while the video is clean. We propose keeping the per-sensor heads and using them to adapt the model on the fly. Each sensor's head is corrected using pseudo-labels (the model's own guesses on unlabeled test data) that come only from the other sensors. A corrupted sensor therefore cannot confirm its own confident mistakes. Only the normalization statistics of an encoder whose inputs have measurably drifted are refreshed. The whole procedure runs without backpropagation, so it applies to CNN and MLP fusion models as well as Transformers.

The opportunity comes from two literatures that have not been joined. Training-time work shows that without a per-modality training signal, a fusion network collapses onto one modality. Test-time adaptation, which updates a deployed model on unlabeled test streams, mostly adapts the fused output as one block. When only one sensor has shifted, that block update damages the clean sensor's features. Our claim is that the per-modality heads left over from training are the right tool for this failure. They separate "which sensor went wrong" from "what class is this," and they are cheap enough to run at batch size one.

Core Claim

Under asymmetric shift, meaning that one modality's input distribution moves while another's does not, a multimodal model can be adapted better and more cheaply than current multimodal test-time methods with three changes. First, it keeps its training-time unimodal heads. Second, it scores each encoder's drift from label-free feature statistics rather than from prediction entropy. Third, it updates each unimodal head's class prototypes using leave-one-modality-out pseudo-labels, and does so without backpropagation. The novel part is the third element: a head is taught only by the other modalities, weighted by how reliable they currently are, while the clean modality's representation stays untouched. A second, separable claim is that heads trained jointly with a balancing objective make better test-time teachers than linear probes fitted after training. If true, this makes the training-time choice part of the robustness story.

Where This Sits

The training-time motivation comes from several papers. What Makes Training Multi-Modal Classification Networks Hard? shows that naive joint training on Kinetics (RGB + audio) reaches 71.7%, below RGB-only at 72.6%. Its Gradient-Blending method, which reweights the per-modality and fused losses by how much each overfits, reaches 75.8%. Related work shows that auxiliary unimodal losses preserve branch quality: Vielzeuf et al., Wu et al., and Bezirganyan et al.. Multimodal Classification via Total Correlation Maximization derives a unimodal term from an information-theoretic objective. Holzbock et al. show that auxiliary losses plus randomly dropping a sensor during training keep radar-only accuracy at 83.0% instead of 78.3% when the other sensor fails. All of these assume the test distribution equals the training distribution. We reuse their training recipe and ask what the heads are worth after the distribution moves.

The closest test-time work falls into three groups.

BriMPR names the "coupling effect," where adapting shared parameters corrupts the clean modality. Its remedy is to tune prompt tokens per encoder so that feature statistics match the source, then apply masked cross-modal recombination and a contrastive loss. It needs Transformer prompt slots and several backward passes, and it never uses task-level unimodal heads. We target the same coupling effect with task heads and no gradients.

SuMi does read unimodal predictions. It selects samples with low fused entropy but higher unimodal entropy, and it aligns unimodal and fused outputs by maximizing mutual information. This alignment is symmetric: every branch is pulled toward every other, including toward the corrupted one. SuMi also still updates the shared trunk by gradient. Our leave-one-out teaching is deliberately asymmetric, and our encoders stay frozen apart from normalization statistics.

MC-TTA comes closest in spirit. It transfers the class geometry of the robust optical-flow stream to the corrupted RGB stream. It does this through teacher–student prototype memory banks with gradient updates, and it assumes in advance which stream is robust. We infer that online, per batch.

A further group constrains the design. Tent minimizes the entropy of the model's predictions. It works on single-modality ImageNet-C, where error falls to 44.0%. Under strong multimodal noise on Kinetics50-C, SuMi reports that Tent falls to 13.1%, against 30.8% for the unadapted model. AEO shows that entropy minimization also erases the ability to flag unknown classes. Together these rule out low entropy as the signal for which sensor to trust, so we use feature-statistic drift instead. AVRobustBench finds that generic methods (TENT, SAR, RPL, EATA, READ, SuMi) gain little under correlated bimodal corruption. Its authors explicitly leave training-time architectural changes unexplored.

The mechanics come from two sources. MemBN keeps a first-in-first-out memory of normalization statistics and uses a Z-score detector to notice shifts, which lets it keep 12.88% error on CIFAR10-C at batch size one. T3A adapts a frozen network by updating class prototypes from confident test features. Both are single-modality. We run one copy of each per sensor and couple them through cross-modal pseudo-labels.

Research Program

We first train source models with auxiliary unimodal heads. We use plain auxiliary losses and a Gradient-Blending-weighted variant, together with sensor dropout in the style of Holzbock et al. We also build two controls. The first uses the same backbone trained without heads, with linear probes fitted afterwards on source features. The second is a naive joint model.

At test time each encoder $m$ keeps its source normalization statistics and a MemBN-style memory of recent test statistics. From these we compute a drift score $s_m$, the average standardized distance between current and source channel statistics. The reliability weight is $r_m = \exp(-s_m/\tau)$, where $\tau$ is a temperature fixed on held-out source-plus-synthetic-corruption data. An encoder refreshes its normalization statistics only when $s_m$ crosses a detection threshold. A clean encoder is therefore left exactly as it was trained.

For each test sample, the pseudo-label for modality $m$ comes from the other modalities only:

$$q_{-m}(y \mid x) \propto \prod_{k \neq m} p_k(y \mid x_k)^{\,r_k},$$

where $p_k$ is head $k$'s class distribution given its own input $x_k$, and a head's influence shrinks as its encoder drifts. When $q_{-m}$ is confident, the feature from encoder $m$ is added to head $m$'s support set for the predicted class. The class prototypes are then recomputed as in T3A. No weights receive gradients.

The final prediction mixes the fusion head with the reliability-weighted product of all unimodal heads. The weight given to the fusion head scales with the smallest $r_m$. The reason is that the fusion head is the component most exposed to a single corrupted input.

The analysis tracks one quantity that current papers do not report: the accuracy of the clean modality's own head before and after adaptation. This is a direct measurement of the coupling effect, and our method should hold it flat by construction.

The drift score also gives a cheap test of AEO's unresolved ambiguity. AEO cannot tell whether cross-modal disagreement reflects a broken sensor or a class the model has never seen. Our separation rule is that disagreement with one drifted encoder suggests corruption, while disagreement with no drifted encoder suggests a novel class. We treat this as a secondary check, not the main claim.

Baselines

The naive reference points are the unadapted source model and the best single modality deployed alone. Both are evaluated on the same source checkpoints as everything else. For comparison, BriMPR reports 31.8% for its source model on Kinetics50-C. Beating the naive points is necessary but not interesting.

The strongest existing method is BriMPR. It reports 40.9% average accuracy on Kinetics50-C and 36.5% on VGGSound-C, where the prior best, READ, reached 32.4%. It is the right competitor because it attacks the same coupling effect with per-modality alignment. We will run the released recipe on our own Transformer source model, with and without auxiliary heads, under the same streams and batch sizes. Hyperparameter tuning will be capped at the same number of held-out-validation trials as our method. We quote the published figure only as a sanity anchor for our reproduction. To count as beating BriMPR, we must match or exceed its average accuracy while showing no drop in clean-head accuracy. We must also show a clear margin on the single-modality-corrupted subsets, using zero backward passes.

SuMi is the strongest method that already reads unimodal predictions. It reports 34.8% on strong out-of-distribution Kinetics50-C, against Tent's 13.1%. It is run the same way. AV2C is the reference for correlated bimodal corruption on VGGSound-2C. The file gives its gains over READ (5.32%) and over SuMi (8.5%), but no reported absolute accuracy was found on Lacuna. We will therefore report our own reproduction.

MC-TTA reports 45.19% on EPIC-Kitchens and 88.61% / 91.07% on UCF→HMDB / HMDB→UCF. It is the baseline for the RGB-plus-flow CNN setting. It matters most there because BriMPR cannot run without prompt slots. We also run Tent applied per branch, which isolates "adapt each modality separately" from our cross-modal teaching.

Three ablations isolate the key ingredient. The first replaces leave-one-out pseudo-labels with each head's own predictions, which is T3A applied per head. The second disables prototype updates and keeps only drift-gated per-modality normalization together with reliability-weighted fusion. The third swaps jointly trained heads for post-hoc probes.

Why not just…?

Why not just refresh each encoder's normalization statistics separately? Mirza et al.'s normalization-only adaptation recovers much of the lost accuracy for free, and doing it per encoder already avoids touching the clean sensor. This is a real threat. On ImageNet-C, though, normalization alone leaves error at 49.9%, against 44.0% when decision boundaries are also adapted (Tent). This suggests statistics alone do not fix the classifier. Our second ablation is exactly this alternative. If it lands within seed noise of the full method, we will conclude that the prototype step does not earn its keep, and we will say so.

Why not just weight the fusion by per-sensor confidence and skip adaptation entirely? Confidence-weighted late fusion is cheap and handles an obviously dead sensor. However, SuMi shows that confidence is misleading under multimodal shift, since low entropy can come from latching onto a corrupted stream. A fixed reweighting also leaves the corrupted modality's head miscalibrated, so it keeps contributing noise. The second ablation, which uses drift-based rather than entropy-based weights, together with an entropy-weighted variant, will show how much of the gain comes from reweighting versus actually repairing the corrupted head.

Why not just use SuMi's mutual-information sharing, which already couples unimodal and fused outputs? Symmetric alignment pulls the clean branch toward the corrupted one as much as the reverse. It also requires gradient updates to the shared trunk. Tracking clean-head accuracy under single-modality corruption tests this directly. If SuMi's clean head does not degrade on these streams, our asymmetry argument weakens to a compute argument. We would report it that way.

Why not just fit probes on an off-the-shelf model instead of retraining with auxiliary heads? This would make the method usable on any public backbone, and it is exactly the third ablation. The training-time literature (Wang et al., Wu et al.) predicts that encoders trained without per-modality supervision carry a weaker signal for the non-dominant modality. That would make probes on those encoders poor teachers. If the prediction fails, the post-hoc variant becomes the recommended one, which would be a useful result in itself.

Proposed Experiments

The main study uses Kinetics50-C and VGGSound-C with an audio-visual Transformer. We split every corruption into video-only, audio-only, and both-corrupted streams. We report average accuracy, accuracy on each subset, clean-head accuracy change, and backward passes and latency per sample at batch sizes 1, 8, and 64.

Correlated bimodal corruption is tested on VGGSound-2C from AVRobustBench across all five severities. The backbone-agnostic claim is tested on a two-stream RGB-plus-flow CNN on UCF–HMDB and EPIC-Kitchens, following MC-TTA's protocol. Continual streams switch corruption type without warning, which tests the drift detector's resets.

The decisive checks are three. First, on single-modality-corrupted streams the full method should beat both per-modality normalization alone and self-pseudo-labeled prototypes, with clean-head accuracy unchanged. Second, jointly trained heads should beat post-hoc probes as teachers. Third, the method should hold its advantage at batch size one, where gradient-based methods are known to be fragile.

If the first check fails, the proposal reduces to "gate normalization by drift per sensor." That is still useful, but it is a much smaller claim, and we would present it as such. As a secondary probe, on EPIC-Kitchens open-set splits we test whether the drift-conditioned disagreement rule separates corrupted known classes from novel classes better than AEO's entropy thresholds, measured by H-score.