Teacher-Labeled Routing: Teaching a Multimodal Student Which Sensors to Run

A model that uses several input streams, such as video frames, motion features and audio, usually runs every stream on every input, even when one stream alone would give the answer. Dynamic modality selection fixes this with a small "router," a cheap network that looks at a quick preview of the input and decides which expensive per-stream networks to run. Today these routers learn poorly. They are trained only from the final task loss plus a compute penalty, through a relaxation of discrete choices (Gumbel-Softmax) whose temperature has to be tuned by hand. They often collapse early onto one fixed choice, and cascades with a fixed order cannot reorder themselves when the usually-cheap stream, audio for instance, is corrupted.

We propose to give the router a direct target, produced by a teacher. A large multimodal teacher sees the clean, complete input. For each training example we measure which subset of the student's streams, run on the degraded input the student will actually receive, reproduces the teacher's prediction most cheaply. That per-example, soft ranking of subsets becomes a label, and the router is trained on it with ordinary cross-entropy. The router then learns two things together: which streams are enough for an example, and which streams to avoid because they are corrupted. A corrupted stream pulls the student's prediction away from the clean teacher, so it ranks badly. Knowledge distillation (training a small model to match a large model's softened outputs) usually supervises what a student predicts. Here it supervises what the student chooses to compute.

Core Claim

A router trained on teacher-derived, cost-aware subset targets will beat routers trained only on task loss. We expect it to reach higher accuracy at matched compute. We also expect it to hold up when the default preview or cascade stream is corrupted, and to need no temperature or top-$k$ annealing schedule. The key ingredient is that the teacher has privileged access to clean, complete data while the student does not. This asymmetry is what lets the target encode "skip this stream, it is unreliable right now." Targets built from ground-truth correctness cannot express that, because the example is usually still classified correctly either way.

Formally, let $x$ be the clean multimodal input and $\tilde{x}$ its degraded version: noise, dropped frames, or missing streams. Let $q(x)$ be the teacher's class distribution. Let $p_S(\tilde{x})$ be the student's prediction when it runs only the subset $S$ of its stream networks, and $c(S)$ the compute cost of $S$ in floating-point operations. The routing target is

$$r(S \mid x) \propto \exp\!\Big(-\big[\mathrm{KL}(q(x)\,\|\,p_S(\tilde{x})) + \lambda\, c(S)\big]/\tau\Big),$$

where $\lambda$ sets the accuracy–compute trade-off and $\tau$ controls how soft the target is. The router, which sees only cheap preview features of $\tilde{x}$, is trained to minimize the cross-entropy between its output and $r$. Here $\tau$ is a fixed label-smoothing constant rather than an annealed relaxation temperature, because gradients never pass through a discrete sample.

Relation to Prior Work

Four papers motivate the gap. AdaMML learns a per-segment policy over RGB, optical flow and audio with Gumbel-Softmax and an efficiency penalty. HCMS runs a fixed audio → appearance → motion cascade gated by confidence thresholds. Neither uses a teacher to supervise the routing decision, and HCMS cannot invert its order when audio is bad. VER documents "early collapse" in shallow routers and treats it with a hand-designed curriculum that gradually narrows the number of active experts. We test whether dense teacher targets remove the need for such schedules. CAT-GS uses calibrated unimodal teachers to steer gating during training, but it does not drop streams at test time. Its reliance on unimodal teachers is a weakness we avoid by using one clean multimodal teacher.

Two papers constrain the design because they are the closest existing combinations of routing and distillation. EgoAdapt distills a multisensory teacher into a student (feature, soft-label and ground-truth losses) and separately learns a Gumbel-Softmax modality policy. The teacher shapes the student's features, not its routing decisions, and the preview modality is fixed. MST-Distill routes between multiple teachers per example during training, but produces a static student that does no selection at test time. Our proposal differs from both: the teacher's output is converted into a supervised label for the student's test-time routing.

Three papers supply methods we reuse. Auxiliary Modality Learning with Generalized Curriculum Distillation shows that a teacher with privileged sensors helps a student lacking them. It assumes those sensors are always absent at test time; we let availability vary per example. DyMo selects recovered modalities at inference by whether they reduce cross-entropy, a per-example sufficiency signal close in spirit to ours. However, DyMo computes that signal at test time through iterative steps and class prototypes. We compute it at training time against a teacher and amortize it into a single router pass. Enumerating all subsets is cheap for two or three streams (at most eight subsets). For larger stream sets we use the greedy subset search justified by the approximate submodularity result in Efficient Modality Selection in Multimodal Learning.

Research Program

We build the student in the style of AdaMML: one stream network per modality, a late-fusion head that accepts any subset, and a light preview network with a recurrent state over video segments. The student's subset predictors $p_S$ are first trained with standard soft-label distillation from the clean teacher under random subset masking. This makes every subset's prediction meaningful before routing labels are computed.

Next we compute $r(S \mid x)$ for each training segment under a corruption curriculum. Audio receives additive noise or dropout, and RGB receives blur, occlusion or dropped frames, each applied independently per segment. The router is trained on these targets. We then fine-tune jointly, recomputing targets every few epochs so they track the student as it improves.

We analyze three questions. First, does the router's choice distribution stay diverse early in training, measured by the entropy of its decisions over epochs, where VER reports collapse? Second, does its ordering flip correctly when the preview stream is corrupted? Third, how sensitive is it to $\lambda$ and $\tau$ compared with the Gumbel temperature schedule of AdaMML?

Baselines

The naive reference is the full-fusion student that always runs every stream, trained with the same teacher distillation. It fixes the accuracy ceiling at maximum compute. Its reported counterpart is the weighted-fusion baseline in AdaMML, relative to which AdaMML claims a 47.3% computation reduction on Kinetics-Sounds.

The strongest existing method is AdaMML itself. On Kinetics-Sounds, AdaMML reports 88.17% accuracy at 76.45 GFLOPs. We re-implement it with our backbones and the same data, epochs and hyperparameter search budget, so that it and our router receive equal tuning trials. We report it both with and without soft-label distillation on its classifier. The distilled variant is an EgoAdapt-style combination and is the fairest test of whether routing supervision, rather than distillation in general, is what helps. Beating it means higher accuracy at matched GFLOPs on clean data, or equal accuracy with at least 15% fewer GFLOPs, together with a clear margin under preview-stream corruption.

The second strong baseline is an HCMS-style confidence cascade. On FCVID, HCMS reports 88.8% mAP at 56% of the full-fusion GFLOPs, and 87.5% mAP at 86% of baseline computation on ActivityNet-v1.3. We run it with thresholds swept on validation data. We also run a stronger variant that picks the best fixed order per corruption type with oracle knowledge of the corruption. This bounds what heuristic ordering can achieve.

For missing streams, the comparison is DyMo on PolyMNIST, which reports 13.12% higher accuracy than the next best method under 80% missingness, averaging 1.38 selection steps per sample. We compare against DyMo on accuracy and on inference cost, since our router needs a single pass.

Two ablations isolate the key ingredients. The first, a ground-truth oracle router, replaces the teacher term in $r$ with the student subset's cross-entropy on the true label. This tests whether a teacher is needed at all. The second, a non-privileged teacher, lets the teacher see the same degraded $\tilde{x}$ as the student. This tests whether clean-input privilege is what teaches the router to skip corrupted streams. A statically distilled single-stream student, as in the MST-Distill setting, marks the low-compute end of the trade-off.

Why not just…?

Why not just use a confidence cascade with a well-chosen order? This is a real threat on clean data, where HCMS already beats full fusion at 56% compute on FCVID. Its weakness is structural: the order is fixed, so if the always-run first stream is corrupted, the cascade either wastes compute on it or exits confidently on bad evidence. The corrupted-preview experiments settle this. If even the corruption-oracle cascade matches our router, the learned router is not worth its complexity and we will say so.

Why not just add distillation to an AdaMML-style Gumbel policy? That is essentially EgoAdapt, and we include it as a baseline. The distinction is where the teacher's signal lands. In that setup the router still learns only from the fused task loss through a relaxed sample, which is the setting where VER observed early collapse and AdaMML needed tuned temperature schedules. If this baseline matches ours in both accuracy and stability across $\tau$ and $\lambda$ sweeps, our contribution reduces to a training convenience.

Why not just label routes by ground-truth correctness? This is the most serious simple alternative, and it costs no teacher. It fails in principle in two ways. A correct-but-degraded subset looks identical to a correct-and-clean one, so the router gets no reason to skip a corrupted stream. And binary correctness is nearly constant on easy examples, which gives little signal. The ground-truth oracle ablation measures this directly. If it closes most of the gap, the idea should be recast as "supervised routing," with distillation as an optional extra.

Why not just distill into one cheap single-stream student? MST-Distill's own RAVDESS result shows static cross-modal students top out low: 66.94% for the baseline student versus 71.74% with its teacher mixture, still a static audio model. MOSEL points out that serving several static models for different trade-offs costs extra GPU memory. Our trade-off curve includes the static student, so the comparison is explicit.

Why not just train static fusion with modality dropout? A Study of Dropout-Induced Modality Bias finds that such dropout makes networks over-rely on particular streams. Even when it works, it saves no compute, because every available stream is still run. It is equivalent to our full-fusion reference with masking, and it appears in our results as that reference.

Proposed Experiments

The main benchmarks are Kinetics-Sounds, ActivityNet-v1.3 and FCVID, the three settings where AdaMML and HCMS report numbers. We evaluate each under clean input and under three corruption regimes: audio corrupted, RGB corrupted, and random per-segment mixes of the two. Metrics are accuracy or mean average precision against GFLOPs per video, reported as full trade-off curves obtained by sweeping $\lambda$. PolyMNIST under 20–80% missingness tests the missing-stream case against DyMo. CREMA-D, where CAT-GS reports 86.29%, serves as a check on a small two-stream emotion recognition task.

Three checks are decisive.

The first is the audio-corrupted curve on Kinetics-Sounds. Our router should invert its default order and approach the corruption-oracle cascade, while AdaMML and the fixed cascade degrade.

The second is the ablation gap. Our router should beat both the ground-truth oracle router and the non-privileged teacher by a margin larger than seed variance over three seeds. That would show that both the teacher and its clean-data privilege matter.

The third is stability. Router decision entropy should stay non-degenerate over the first epochs, and final accuracy should vary less across the $\lambda$ and $\tau$ grid than AdaMML's does across its temperature grid.

If the first check fails, the corruption-awareness claim fails. If the second fails, the distillation claim fails. We will report whichever outcome occurs.