Many deployed systems see only one cheap sensor, while training data often arrives with extra streams: RGB video alongside skeleton joints, LiDAR alongside cameras, audio and video alongside transcripts. A natural way to use large unlabeled multi-sensor collections is to train a multimodal "teacher" on the few labeled examples, let it label the unlabeled pool, and train a small single-sensor "student" on those labels. The student then runs alone at test time and pays none of the multimodal cost. This matters because MultiBench, a suite of 15 multimodal datasets, found that cross-attention models such as MulT need more than 12× the inference time of unimodal baselines.
The weak point is deciding which teacher labels the student should trust. A teacher's confidence tells you whether its label is probably correct. It does not tell you whether the student could ever reproduce that label from the one sensor it will have. We propose to score each pseudo-label along both axes. The second score comes from a counterfactual: run the same teacher a second time with its access to the auxiliary sensor switched off, and measure how far its prediction moves. Labels that are confident and depend only on the cheap sensor, labels that the auxiliary sensor corrected, and labels that exist only because of the auxiliary sensor are different kinds of supervision. The student should treat them differently.
Core Claim
A multimodal teacher's fused confidence is a poor predictor of whether a pseudo-label helps a student that sees only one modality. A counterfactual gap, meaning the change in the teacher's prediction when the auxiliary modality is removed, supplies the missing axis of reliability. This mirrors a unimodal finding. Decoupled Pseudo-Labeling for Semi-Supervised Monocular 3D Object Detection measured a Pearson correlation of only −0.196 between 2D classification confidence and 3D depth error, and responded by filtering the two attributes with separate criteria. We expect an analogous split between "correct" and "learnable from the deployed sensor," and we propose to test it directly before building on it.
Concretely, the teacher is a gated cross-modal attention model. Cross-modal attention here means that features from the cheap modality issue queries over features from the auxiliary modality. A learned gate $g \in [0,1]$ mixes the attended context back into the cheap-modality features, following the gating design of An Interpretable Framework for Drug-Target Interaction with Gated Cross Attention. Setting $g=0$ gives a clean "cheap-only" forward pass through the same weights. The cost is one extra teacher pass per unlabeled sample, which is small next to training two peer networks.
For an unlabeled sample $x$:
- $p_f(x)$ is the fused teacher prediction.
- $p_0(x)$ is the prediction with the gate closed.
- $c_f(x)=\max_k p_f(x)_k$ is the fused confidence.
- $\Delta(x)=\mathrm{KL}(p_f(x)\,\|\,p_0(x))$ is the counterfactual gap.
The student's loss weight on the hard pseudo-label $\arg\max_k p_f(x)_k$ is
$$ w(x) = \mathbb{1}[c_f(x) > \tau]\; h\big(\Delta(x)\big), $$
where $\tau$ is a confidence threshold. The function $h$ is either one of three fixed policies (keep all, drop high-gap, soft down-weight) or a monotone function fit on a small labeled validation split. Which shape of $h$ is right is an empirical question. High-gap samples may carry the most useful privileged information, or they may be targets the cheap sensor cannot support. The proposal is designed to answer that question rather than assume the answer.
The labeling is decoupled in the sense surveyed in the research file. The teacher is frozen after its labeled-data training and never consumes the student's outputs, so the student cannot reinforce its own errors. The pseudo-labels are computed once offline, following the static labeling used by Large Self-Supervised Models Bridge the Gap in Domain Adaptive Object Detection, which replaced a moving-average teacher with a frozen DINOv2-based labeler.
Relation to Prior Work
The closest work splits pseudo-labeling into prediction and reliability scoring. DREAM predicts labels by label propagation and scores their reliability with a separate cross-attention module for semi-supervised 2D-3D retrieval. In DREAM, both modalities are present at test time, and reliability means correctness. We score learnability for a student that loses one modality, and we derive the score from a counterfactual pass rather than a trained scorer. Cross Pseudo-Labeling for Semi-Supervised Audio-Visual Source Localization fights confirmation bias with two multimodal peer networks that label each other. This doubles training compute and keeps both modalities at inference, whereas our student is single-modality and the teacher is labeled once. ATSO supplies the decoupling principle for the unimodal self-training baseline: its models label disjoint data subsets, and it resets the student between generations.
A second group removes heavy modalities at test time, but only under full supervision. Multi-Modality Co-Learning for Efficient Skeleton-based Action Recognition (MMCL) uses RGB and a multimodal LLM as training-only teachers for a skeleton graph network. LabelDistill distills ground-truth-derived features into a camera detector and adds zero inference cost. Both motivate the deployment regime we target. Neither uses unlabeled data, so neither faces the question of which teacher labels to trust. Modality-specific Distillation shows that decomposing distillation per modality beats monolithic distillation. It supplies our soft-distillation baseline, and it is the reason we expect per-modality attribution to matter. CA²ST connects frozen unimodal experts with bottleneck cross-attention, and we reuse that design as the teacher's fusion block. CA²ST itself remains fully supervised and runs three backbones at inference.
One result constrains the design. MultiBench reports that simple late fusion often gives the best efficiency-accuracy trade-off. This is why a late-fusion teacher appears below as a first-class ablation rather than a strawman.
Research Program
The work has three stages.
The first stage is diagnostic. On a benchmark with labeled and unlabeled multimodal data, we train the gated cross-attention teacher on the labeled split. For each unlabeled sample we then estimate how much its pseudo-label helps a cheap-modality student. We estimate this by influence approximations checked against small leave-group-out retraining runs over bins of samples. We then measure how well $c_f$ and $\Delta$ each predict that helpfulness. If $\Delta$ adds no predictive power beyond $c_f$, the proposal stops at this stage and reports the negative result.
The second stage builds the weighting rule and trains students under a fixed pseudo-label budget. The three fixed $h$ policies are compared with the validation-fit monotone $h$.
The third stage asks whether cross-modal attention earns its cost in the teacher. We swap the teacher for a late-fusion model, in which removing the auxiliary branch gives the counterfactual, and for token-level variants. In the token-level variants, the cross-attention map localizes which cheap-modality tokens the auxiliary stream changed. This matters for dense tasks. The primary setting is skeleton action recognition on NTU RGB+D 120 with skeleton as the deployed modality and RGB as the auxiliary. The secondary setting is CMU-MOSEI with text as the deployed modality and audio and video as auxiliary, which tests transfer to a regime where MultiBench documented the fusion overhead.
Baselines
The naive reference is the skeleton-only graph network (CTR-GCN, the backbone MMCL builds on) trained on the labeled split alone. It sets the floor, and every method must beat it at identical inference cost.
The unimodal self-training baseline trains the same network on the unlabeled pool with its own confidence-thresholded pseudo-labels. It uses ATSO's disjoint-subset cross-labeling and resets so that it is a decoupled baseline rather than naive self-training. It is the strongest competitor that uses no extra modality. If it matches the proposal, then the auxiliary sensor is not worth collecting.
The fused-confidence baseline uses exactly our teacher and threshold $\tau$ but sets $h \equiv 1$. This is offline frozen-teacher labeling in the style of DINO Teacher, applied cross-modally. It is the ablation that isolates the counterfactual gap, the proposal's key ingredient.
The soft-distillation baseline trains the student on the teacher's full softened distribution over all unlabeled samples, with modality-specific auxiliary losses following Modality-specific Distillation. It is the strongest existing method on this question, because soft targets sidestep thresholding altogether. MMCL-style supervised co-learning on the labeled split is added on top of every method, so that we test pseudo-labeling on top of MMCL rather than instead of it.
For reference, MMCL reports top-1 accuracy on NTU-120 cross-subject rising from 88.9% to 90.3% with full labels. There is no reported result on Lacuna for semi-supervised NTU-120 at 10% labels, so we will establish all low-label numbers ourselves. On Hateful-Memes, Modality-specific Distillation reports 68.30% AUC against 66.53% for conventional distillation, which calibrates the size of gains distillation variants typically deliver.
Fairness is enforced on several fronts. All methods share the labeled and unlabeled splits (10% and 5% labeled), the student architecture, the augmentation, and the student training schedule. Each method receives the same hyperparameter search budget, 20 trials tuned on the same held-out labeled validation set, including $\tau$ for the thresholded methods and the temperature for distillation. We report teacher compute separately: two passes for our method, one for the fused and distillation baselines, none for self-training.
Beating the strongest baseline means exceeding both the fused-confidence and soft-distillation students by more than the seed-to-seed standard deviation across five seeds, at both label fractions, with identical student inference FLOPs.
Why not just…?
Why not just distill the teacher's soft outputs on all unlabeled data? This is the most serious threat. Soft distillation already down-weights uncertain samples implicitly. It does not, however, distinguish a confident fused prediction that relies entirely on RGB from one the skeleton stream supports. Our diagnostic stage measures exactly that distinction. We also run a hybrid in which $h(\Delta)$ reweights the soft distillation loss. If plain soft distillation matches our method and the hybrid adds nothing, we will report that the counterfactual signal is redundant with soft targets.
Why not just self-train the cheap modality without any teacher? Unimodal self-training avoids teacher cost entirely, and ATSO shows it can be strong: 81.69% Dice on NIH Pancreas CT at 10% labels, against 78.98% for plain self-learning. But its pseudo-labels are bounded by what the skeleton model already knows. The ambiguous classes where MMCL's RGB guidance helped most, such as "Writing," which gained nearly 8%, are the ones we expect it to miss. Per-class comparison on those classes settles this.
Why not just use a late-fusion teacher, given MultiBench's finding? Late fusion also permits a counterfactual, by dropping the auxiliary branch, and it is cheaper. So it may well suffice for clip-level classification, and we treat that outcome as plausible. The claim we defend is the counterfactual reliability score. Cross-attention is the default teacher only because its gate gives an in-network counterfactual and token-level attribution. If late fusion matches on NTU and MOSEI, we will recommend it and confine cross-attention to dense tasks.
Why not just raise the confidence threshold? A stricter $\tau$ removes low-confidence noise. It cannot remove confident labels that are unsupportable from the cheap sensor, because those labels are confident by construction. The fused-confidence baseline swept over $\tau$ within the same tuning budget tests this directly.
Proposed Experiments
The first decisive check is the diagnostic correlation. On NTU-120 at 10% labels we compute, per unlabeled bin, the student-side helpfulness and its rank correlation with $c_f$ and with $\Delta$ conditional on $c_f$. The proposal proceeds only if $\Delta$ explains significant additional variance.
The second check is the main comparison. We report top-1 accuracy on NTU-120 cross-subject and cross-setup splits, and Acc-2, F1 and mean absolute error on CMU-MOSEI with a text-only student. These are reported at 5% and 10% labels against all baselines with five seeds, together with pseudo-label precision in each $\Delta$ bin.
The third check is the shape of $h$. Whether the best policy keeps, drops, or down-weights high-gap labels is the scientific result, and it tells practitioners whether auxiliary-only evidence transfers to a cheap sensor.
The fourth check is cost accounting. Student inference latency and parameters must equal the skeleton-only baseline exactly, and teacher labeling cost is reported in GPU-hours against the doubled training cost of peer teaching.
The fifth check is a stress test. We corrupt the auxiliary modality on a fraction of unlabeled samples. A useful reliability score should show inflated $\Delta$ on those samples and protect the student, whereas fused-confidence filtering should degrade.
References
- MultiBench: Multiscale Benchmarks for Multimodal Representation Learning
- Decoupled Pseudo-labeling for Semi-Supervised Monocular 3D Object Detection
- An Interpretable Framework for Drug-Target Interaction with Gated Cross Attention
- Large Self-Supervised Models Bridge the Gap in Domain Adaptive Object Detection
- DREAM: Decoupled Discriminative Learning with Bigraph-aware Alignment for Semi-supervised 2D-3D Cross-modal Retrieval
- Cross Pseudo-Labeling for Semi-Supervised Audio-Visual Source Localization
- ATSO: Asynchronous Teacher-Student Optimization for Semi-Supervised Image Segmentation
- Multi-Modality Co-Learning for Efficient Skeleton-based Action Recognition
- LabelDistill: Label-Guided Cross-Modal Knowledge Distillation for Camera-Based 3D Object Detection
- Modality-specific Distillation
- ${\text{CA}^{2}\text{ST}}$: Cross-Attention in Audio, Space, and Time for Holistic Video Recognition