Spend the Label or Fetch the Sensor? Uncertainty Decomposition for Active Adaptation with Missing Modalities

Consider a multimodal model moved to a new setting: a radar-plus-camera activity recognizer installed in a new building, or a meme classifier applied to a new topic. Labels are scarce, and in many incoming samples one input stream is absent because a sensor failed or a field was never collected. Active learning, meaning the model chooses which unlabeled samples a human should annotate, is the standard response to label scarcity. When inputs are incomplete, though, it breaks in a specific way. A sample with a missing stream looks uncertain largely because the stream is missing. Uncertainty-based selection therefore spends the budget on samples whose ambiguity a label cannot resolve.

We propose to split a model's uncertainty on an incomplete sample into two parts. The first is the part caused by not knowing what the missing modality would have looked like. The second is the part that remains even if that modality were known. The split also tells us which of two actions to buy. If uncertainty is driven by the missing stream, the useful action is to acquire that stream (re-measure, request the scan, fetch the caption). If it remains when the stream is known, the useful action is to acquire the label. Samples that are completed this way also become paired target-domain data, which re-anchors the cross-modal reconstructor to the new domain, precisely where source-trained reconstructors are known to fail.

Core Claim

Let $x_o$ be the observed modalities of a target sample and $z$ the latent features of the missing modality. A stochastic cross-modal reconstructor $q(z \mid x_o)$ produces samples of plausible missing features. Here "cross-modal feature reconstruction" means predicting the embedding of an absent stream from the present ones. Writing $y$ for the task label, the classifier's predictive entropy on $x_o$ decomposes exactly as

$$ H[y \mid x_o] \;=\; \mathbb{E}_{z \sim q(z\mid x_o)}\, H[y \mid x_o, z] \;+\; I(y; z \mid x_o). $$

The second term, the mutual information between label and reconstruction, measures how much the prediction would change depending on which plausible reconstruction is true. Observing the real modality removes it for that sample. The first term is uncertainty that persists given complete inputs. We further split it with Monte Carlo dropout over the classifier's adapter weights into a model-ignorance part, which labels reduce, and a residual part, which nothing reduces:

$$ \mathbb{E}_{z} H[y \mid x_o, z] \;=\; \underbrace{\mathbb{E}_{z}\, I(y;\theta \mid x_o, z)}_{\text{reducible by a label}} \;+\; \mathbb{E}_{z,\theta}\, H[y \mid x_o, z, \theta], $$

where $\theta$ denotes the adapter weights. Each round, the acquisition rule ranks sample–action pairs by reducible uncertainty per unit cost. Completion is scored by $I(y;z\mid x_o)/c_{\text{mod}}$ and labeling by the label-reducible term divided by $c_{\text{label}}$. The rule also deliberately skips samples dominated by the irreducible term. The claim is that, at a fixed total budget, this two-action, decomposition-aware policy beats label-only active domain adaptation on incomplete target data, and that its advantage grows with the missing rate and the domain gap.

Where This Sits

Three lines of work each cover two of the three pieces. Active domain adaptation methods assume complete inputs. mmADA uses evidential, Rényi-entropy uncertainty to separate domain uncertainty from prediction uncertainty and labels 5% of target radar data, but every radar view is assumed present. DAM pairs active queries with CLIP prompt tuning for source-free adaptation, but its target tasks are unimodal. These methods motivate our setting and supply the uncertainty toolkit. Our addition is a separate term for uncertainty caused by absent inputs and a second purchasable action.

Missing-modality adaptation methods work in passive regimes. MoRA adapts CLIP with shared and modality-specific low-rank weights and fills gaps with placeholder inputs. AOEPT compensates missing modalities by querying prompt prototypes built from complete data. Continual missing-modality prompting indexes prompts by which modalities are present. None of them chooses which data to acquire. AOEPT's own analysis (art_08b73182; AOEPT: Breaking the Implicit Modality-Reduction Bottleneck in Modality-Missing Prompt Tuning) warns that its prototype repository assumes training and test distributions match. That warning constrains our design: the reconstructor must be updated on the target domain, which is exactly what the completion action pays for.

Reconstruction-with-uncertainty methods are in-domain and passive. UMQ estimates modality quality by ranking and enhances degraded features. UMR turns embeddings into Gaussians whose variance re-weights fusion. CyIN and MD2N train cyclic or diffusion translators on complete source data. We reuse the reconstruction machinery from this line, specifically the residual-autoencoder translator of CRA as implemented in the CyIN codebase, and make it stochastic. Unlike these methods, we use its uncertainty to decide acquisitions under a shift.

One cited result appears to contradict our design. ALFLTP finds that selecting median rather than maximum aleatoric uncertainty is better, because the highest-uncertainty samples are unlearnable noise. A naive reading says severely incomplete samples should be discarded. Our decomposition refines this rather than rejecting it. Missing-stream uncertainty looks irreducible to a label-only learner, but it is reducible by a different action. Only the residual term deserves ALFLTP-style filtering.

Research Program

We will build a pipeline with three components. The first is a frozen multimodal backbone adapted with low-rank adapters in the style of MoRA, so that adaptation from tiny budgets does not overwrite pretrained knowledge. The second is a small residual translator per missing-modality pattern, made stochastic by a deep ensemble of five heads plus a predicted diagonal variance. The third is MC-dropout on the adapters. Computing the decomposition then costs $K$ translator samples times a few classifier-head passes per unlabeled sample. The backbone forward pass is shared, so scoring stays cheap. This matters given the cost concerns raised by SIFT.

After each acquisition round, labeled samples update the adapters. Completed samples give paired target features, which update the translator with a reconstruction loss plus the cycle-consistency term from CyIN. Completed samples can also be labeled later, and the policy is free to buy both actions for one sample.

A risk is that the translator becomes overconfident under shift. In that case $I(y;z\mid x_o)$ underestimates the value of completion. We guard against this in two ways. The ensemble disagreement captures translator ignorance, not only spread within a single head. We also measure translator calibration on held-out target pairs, reporting the coverage of true features by the predicted distribution, throughout training.

We will study two regimes. In the recoverable regime, both actions are available, with cost ratios $c_{\text{mod}}/c_{\text{label}} \in \{0.25, 1, 4\}$. In the irrecoverable regime, missing streams can never be fetched. The decomposition then serves only to keep label-only selection away from samples whose uncertainty is driven by the missing stream, while completed source pairs keep training the translator.

Baselines

The naive reference point is random acquisition of labels, with zero-filled missing modalities, adapted by the same low-rank adapters. Any method that cannot beat it at matched budget is not worth its complexity.

The strongest existing competitor for active adaptation is mmADA art_368c9f4d. It is designed for the same label-scarce, multi-sensor shift problem, and its Rényi evidential uncertainty already separates domain from prediction uncertainty. We will run it unchanged on the observed modalities, with missing ones zero-filled, and also with our deterministic translator filling the gap. This gives it every reconstruction advantage except the decomposition. On the vision–language benchmarks, the matching strongest competitor is DAM art_0c6b1dd1 (DAM: Dual Active Learning with Multimodal Foundation Model for Source-Free Domain Adaptation) run on the CLIP backbone. The strongest passive missing-modality competitor is MoRA art_365aad57 (MoRA: Missing Modality Low-Rank Adaptation for Visual Recognition) trained on randomly acquired labels.

The reported reference points from the file are as follows. On complete-modality MM-Fi, mmADA reports 96.88% mean accuracy against 89.90% for EADA (art_368c9f4d). On Office-Home, DAM reports 82.0% accuracy (art_0c6b1dd1; DAM: Dual Active Learning with Multimodal Foundation Model for Source-Free Domain Adaptation). We found no reported result on Lacuna for any active method on MM-Fi with missing modalities, or for any method on Hateful Memes transferred to Harmful-C/Harmful-P, so we will establish these numbers ourselves.

Fairness rules apply to every method. All methods share the backbone, adapter rank, translator architecture where one is used, total budget in cost units, round schedule, and hyperparameter search budget (the same number of trials on a source-only validation split). Label-only methods receive their entire budget as labels.

Two ablations isolate the key ingredients. The first keeps both actions but replaces the decomposition with total predictive entropy, choosing an action by a fixed rule (complete if a stream is missing). The second keeps the decomposition but allows only labels. Beating mmADA means a gain at matched budget larger than the seed-to-seed standard deviation across five seeds, at missing rates of at least 50%. We also expect the gap to widen with missing rate. A flat or shrinking gap would count against the claim even if the average is positive.

Why not just…?

Why not just zero-fill the missing stream and run mmADA or DAM? The file's evidence says zero-filling damages the joint representation. AOEPT shows models trained on what is left collapse into a unimodal subspace, and CRA reports up to a 21% relative recognition margin for imputed over partial inputs. Neither result tests active selection, though. Our direct test is the zero-filled mmADA baseline. We will also measure what fraction of its queries land on incomplete samples and how much accuracy each such label buys compared with a label on a complete sample.

Why not just restrict queries to target samples that happen to be complete? This is the most serious threat. It is simple, avoids the reconstruction-driven entropy problem entirely, and at low missing rates it will likely match us. Its weakness is that at 70–90% missingness complete samples are rare and may be unrepresentative of the target domain. It also never improves the translator, which still serves every incomplete test sample. We run it as a baseline at every missing rate. If it matches us at high missingness, the decomposition is not needed and we will report that.

Why not just use total uncertainty from MC dropout through the whole pipeline (plain BALD)? Total epistemic uncertainty mixes translator ignorance with classifier ignorance. It can rank samples, but it cannot say which action to buy. Tsymbalov et al. (art_6eab1748; Dropout-Based Active Learning for Regression) support model-uncertainty sampling in general, so this alternative may perform well in the label-only regime. The first ablation measures exactly this. If it ties the decomposition in both regimes, the extra term is unjustified.

Why not just pre-train a strong generative imputer such as MD2N on source data and keep it frozen? A frozen imputer cannot follow a target shift, as AOEPT's stated limitation and the file's "distribution gap" concern both indicate. Multi-step diffusion is also expensive at scoring time. We test this alternative by swapping our translator for a frozen source-trained one and tracking target reconstruction error and calibration across rounds.

Proposed Experiments

The primary testbed is MM-Fi under mmADA's cross-environment and cross-user splits. Missingness is simulated by dropping the RGB-D stream or one radar view at rates of 30%, 50%, 70%, and 90%, with completion revealing the held-back stream. XRF55 serves as a second radar benchmark. The vision–language testbed trains on Hateful Memes (benchmark) and adapts to Harmful-C, Harmful-P, and Offensive, with text or image missing, measured by AUROC. A sentiment testbed, with CMU-MOSI as source and CMU-MOSEI as target and missing audio or video, measured by Acc2, probes a milder shift. Budgets sweep from 1% to 10% of the target pool in cost units, and we report area under the budget–accuracy curve as well as final accuracy.

Two checks decide whether the mechanism is real rather than merely helpful. First, on held-out incomplete target samples, we reveal the true modality and measure the drop in the classifier's entropy. This drop should rank-correlate with the predicted $I(y;z\mid x_o)$ well above total entropy's correlation. If it does not, the decomposition is not measuring what we claim. Second, the translator's calibration on target pairs should improve over rounds under our policy but not under label-only policies. Most of our gain over completed-only selection should then appear on incomplete test samples. If the gain instead concentrates on complete test samples, the benefit comes from better labeling, not from reconstruction. Finally, sweeping the cost ratio should shift the policy's action mix in the expected direction, with the advantage shrinking toward the label-only ablation as completion becomes expensive.