Multimodal models get some of their best predictions from synergy: information that exists only when two modalities are read together. Sarcasm is a typical case, where the words say one thing and the tone says the opposite. A separate line of interpretability research shows that a pattern which fires only when many input pieces line up at once becomes exponentially less stable as the number of pieces grows, so a small change to any one piece can break it. Put these together and you get a prediction nobody has tested. When a multimodal model is moved to a new domain (new speakers, new scanners, noisier microphones), its synergistic knowledge should be the first thing to fail. Standard domain-adaptation tools would then be misdirected: they push the whole fused representation toward invariance, and they take their class guidance from a classifier that depends on that fragile synergy.
We propose to measure this failure directly and then adapt around it. The model is a mixture of experts, meaning several sub-networks combined by a small learned "router" that weights each one per example. Each expert is trained to capture one kind of multimodal information: unique to one modality, redundant across modalities, or synergistic. Rather than one domain discriminator on the fused features, we give each expert its own adversarial alignment. The synergy expert's alignment is conditioned on predictions from the stable redundancy expert instead of the model's own output. The router is also penalized for leaning on the synergy expert on target inputs where that expert is measurably unstable. The two-sentence version: synergy is a high-order interaction and should break first under domain shift, so we align each kind of interaction separately and borrow class guidance from the stable experts when aligning the fragile one.
Core Claim
We make two claims, ordered so that the first can be true even if the second fails.
The diagnostic claim is about where the damage goes. Under realistic multimodal domain shift, the accuracy drop of an interaction-typed mixture of experts is concentrated in the synergy expert. The synergy expert's salient interactions are higher-order and cross-modal, and their utilities change far more between source and target than those of the uniqueness and redundancy experts.
The method claim is that order-aware alignment beats whole-vector alignment on target accuracy. Order-aware alignment has three parts: per-expert discriminators, conditioning anchored on the redundancy expert, and a stability-penalized router. It should also keep most of the in-domain synergy benefit, which simply throwing away the synergy expert would not.
Grounding in Prior Work
The stability side comes from the game-theoretic interaction literature. Explaining Generalization Power of a DNN Using Interactive Concepts defines concepts as Harsanyi interactions. A Harsanyi interaction is the extra contribution a set of input variables makes only when all of them are present together. The paper proves their variance under input noise grows exponentially with order and finds that 80–95% of possible interactions are near zero. Towards a Unified Game-Theoretic View of Adversarial Perturbations and Robustness shows that adversarial attacks flip high-order interactions while largely sparing low-order ones. Extracting Robust On-Manifold Interactions Encoded by Neural Networks finds high-order interactions are disproportionately destabilized by off-manifold perturbations. These three papers motivate the proposal, but all of them study unimodal inputs under additive noise or attack, never cross-modal interactions under domain shift. We supply that missing case.
The expert side comes from two papers that decompose multimodal information into experts. I2MoE builds one expert per modality plus a redundancy expert and a synergy expert. It trains them with masking-based losses, for example penalizing the synergy expert when it succeeds from a single modality, and reports gains over a vanilla multimodal transformer (+5.5% accuracy on ADNI, +3% on MOSI). Mixture of Multimodal Interaction Experts sorts examples offline into agreement, disagreement and synergy regimes and reaches 74.93 F1 on MUSTARD, with very large gains on synergy subsets. Both are evaluated only in-domain. Neither has a domain-adaptation mechanism, and neither measures interaction order. I2MoE supplies our architecture. MMOE supplies evidence that synergy is too valuable to discard.
The adaptation side comes from Conditional Adversarial Domain Adaptation (CDAN). CDAN conditions the domain discriminator on the outer product of features and class predictions to avoid aligning the wrong classes. It reports 87.7% on Office-31 against 82.2% for unconditioned adversarial adaptation, and 65.8% on Office-Home against prior methods around 57–58%. CDAN supplies our method and also our main competitor. Its conditioning signal is the full model's prediction, which under our hypothesis is contaminated by fragile synergy on the target.
Cross-modal adversarial adaptation exists in CMDA, which uses camera features to bridge LiDAR domains, but CMDA applies one discriminator on a dense backbone. Multi-Source Domain Adaptation with Mixture of Experts assigns experts to source domains rather than to interaction types.
Sparse Mixture-of-Experts are Domain Generalizable Learners (GMoE) shows that routed experts improve out-of-distribution accuracy (48.7% on DomainNet against 47.3% for a plain ViT). GMoE complicates our story somewhat: modular routing in late layers helps rather than hurts robustness. Our reading is that GMoE's experts specialize on part-level attributes, which are plausibly lower-order. An expert built specifically to be synergistic is the opposite design. Phase 1 tests that reading by measuring order per expert.
Research Program
Phase 1: measuring interaction order per expert. We split each multimodal input into a small number of variables, for example four transcript segments, four audio windows and four video windows on MOSI and MUSTARD, giving $n=12$. Because $n$ is small, Harsanyi interactions can be computed exactly over all $2^{12}=4096$ coalitions. This avoids the sampling error that limits most interaction studies. For an expert's output $v$ evaluated with only the variables in $T$ kept (the rest set to a fixed baseline), the interaction of a variable set $S$ is
$$I(S)=\sum_{T\subseteq S}(-1)^{|S|-|T|}\,v(T),$$
and its order is $|S|$. We call $I(S)$ cross-modal when $S$ contains variables from at least two modalities.
For each expert of a trained I2MoE we record three things on source and on target data. The first is the order distribution of salient interactions. The second is the fraction of salient interactions that are cross-modal. The third is a drift score: the normalized change in interaction utilities between matched source and target conditions. For the corruption shifts in Phase 3 this is the same example clean and corrupted; for natural shifts it is the class-conditional distribution of salient interactions. The diagnostic claim predicts that the synergy expert has the highest mean order and the largest drift, and that across experts and shifts, drift predicts the per-expert accuracy drop.
Phase 2: order-aware adaptation. Each expert $e$ gets its own domain discriminator, trained through a gradient reversal layer. A gradient reversal layer passes features forward unchanged but flips the sign of the gradient on the way back, so the feature extractor learns to fool the discriminator. The uniqueness and redundancy experts use plain feature-level alignment, because their interactions are expected to be low-order and stable. The synergy expert uses conditional alignment, with the conditioning taken from the redundancy expert:
$$z_{\text{syn}}(x)=h_{\text{syn}}(x)\otimes p_{\text{red}}(x),$$
where $h_{\text{syn}}$ is the synergy expert's representation and $p_{\text{red}}$ is the redundancy expert's softmax prediction. In this design the synergy features are class-matched across domains using labels from the expert least likely to be corrupted by the shift. The randomized multilinear map from CDAN keeps the outer product tractable.
The router is trained on target data with an added penalty $\lambda\, r_{\text{syn}}(x)\, s(x)$. Here $r_{\text{syn}}$ is the weight the router gives the synergy expert, and $s(x)$ is a cheap stability estimate: the variance of the synergy expert's logit under random masking of a few input variables. This is a proxy for high-order fragility, motivated by the exponential-variance result. The penalty asks the router to rely on synergy only where synergy is locally stable.
Phase 3: shifts and data. The main testbeds are three, chosen from datasets I2MoE and MMOE already use. On MUSTARD we move from the speaker-dependent split used by MMOE to a speaker-independent split, so the target speakers are unseen. On MOSI we use controlled sensor shifts: additive audio noise at graded SNRs, video blur or frame drop, and transcripts with simulated recognition errors, each at several severities so drift can be plotted against severity. On ADNI we split by acquisition site, if site metadata is available in the release we obtain; if not, we fall back to graded MRI corruption.
As an optional stretch on a different scale, we would run the per-expert discriminators with redundancy anchoring inside a camera-plus-LiDAR detector, using CMDA's nuScenes→KITTI setting.
Baselines
The naive reference point is source-only I2MoE with no target-domain training, alongside a source-only vanilla multimodal transformer (MulT). This fixes the size of the domain gap. No reported result found on Lacuna for either model under any of our shifts, so we measure these ourselves. In-domain, the file records I2MoE at +5.5% over MulT on ADNI and +3% on MOSI.
The strongest existing competitor is CDAN with entropy weighting (Conditional Adversarial Domain Adaptation), applied to the fused output of the same I2MoE backbone and conditioned on the full model's predictions. It is the strongest because it already fixes class-mismatched alignment, the main failure of unconditioned adversarial adaptation. The file reports 87.7% average accuracy on Office-31 and 65.8% on Office-Home for CDAN+E. No reported result found on Lacuna for CDAN on any multimodal shift benchmark.
We also run unconditioned whole-vector adversarial alignment, the gap CDAN closed (82.2% on Office-31 in the CDAN paper). On MUSTARD only, as an in-domain-strong reference, we include MMOE's offline regime routing, which reports 74.93 F1 speaker-dependent (Mixture of Multimodal Interaction Experts).
All methods share the same encoders, expert count, unlabeled target pool, epoch budget and number of hyperparameter trials, 20 random-search trials each. Selection uses the same small labeled target validation set, reported separately as an oracle-selection caveat. Every number is averaged over five seeds.
Beating CDAN means a mean target gain at least twice the seed standard deviation on at least two of the three shifts. It also means in-domain accuracy within one point of source-only I2MoE.
The ablations isolate each ingredient. The first uses per-expert discriminators with no anchoring, where the synergy expert is conditioned on its own predictions. The second uses anchoring with a single shared discriminator. The third removes the stability penalty from the router. The fourth anchors on a separately trained unimodal classifier instead of the redundancy expert, which tests whether "stable anchor" or specifically "redundancy" is what matters.
Why not just…?
Why not just run CDAN on the fused features? It is the obvious move, and it is the main baseline. Our objection is specific and testable. CDAN's conditioning uses the full model's prediction. If Phase 1 shows the synergy expert drifting most, that prediction is least reliable exactly on the target examples where class matching matters. The comparison between redundancy-anchored and self-conditioned per-expert alignment, the first ablation, settles whether the anchor matters. If self-conditioning does as well, the anchoring idea is wrong, and the contribution shrinks to the diagnostic.
Why not just drop or down-weight the synergy expert on the target, or train with modality dropout? This is the real threat. The game-theoretic robustness paper shows that blunt suppression of high-order interactions, as with Cutout, buys robustness (Ren et al.). Against this, MMOE's large synergy-subset gains suggest discarding synergy forfeits real accuracy on exactly the hardest examples. We run three versions of this alternative: synergy weight zeroed at test time, a fixed global down-weighting tuned on target validation, and I2MoE trained with modality dropout. We report results on the full target set and on the target examples MMOE's criterion labels as synergistic. If any of the three matches our method on both slices, we conclude that fragility should be handled by suppression rather than alignment, and we say so.
Why not just reweight the router at test time by prediction entropy, with no adversary? Entropy-based reweighting is cheap and needs no discriminators. Its weakness is that it cannot distinguish an expert that is confidently wrong under shift from one that is right. High-order interactions are exactly the kind that can flip sign while staying confident: Ren et al. report order-$\approx 0.95n$ interactions turning negative under attack. We include entropy reweighting as a baseline and compare its synergy weights with our masking-variance score. The test is which signal better predicts per-example synergy-expert errors, measured by AUROC.
Why not just assign experts to domains, as in multi-source MoE adaptation? Multi-Source Domain Adaptation with Mixture of Experts needs several labeled source domains and gives no handle on which kind of multimodal information breaks. Our setting has one source. On ADNI, where multiple sites can serve as sources, we run it as a secondary baseline.
Proposed Experiments and Decisive Checks
The first check is diagnostic and stands on its own. Across the graded MOSI corruptions, the synergy expert's drift score should rise faster with severity than the other experts' drift. Across all expert-and-shift pairs, drift should correlate with the per-expert accuracy drop with Spearman correlation above 0.5. If the synergy expert is no higher-order than the redundancy expert, or does not drift more, the premise is false and Phase 2 is not worth running. That would still be a useful negative result, given GMoE's finding that routed experts help out of distribution.
The second check is the head-to-head against CDAN on fused features under the matched protocol above. The third separates the proposal from the cheap alternatives. Our method must beat synergy dropping and modality dropout specifically on the synergy-labeled target slice while matching them on the rest. Otherwise the alignment machinery is not earning its cost.
Finally, we recompute the Phase 1 order distributions after adaptation. Our explanation predicts that order-aware adaptation lowers the synergy expert's drift without collapsing its interactions to low order, meaning it keeps its cross-modal fraction. Whole-vector CDAN, by contrast, should either leave drift unchanged or erase cross-modal interactions altogether. If these mechanistic predictions fail while accuracy improves, the method works for a reason we have not identified, and we would report it that way.
References
- Explaining Generalization Power of a DNN Using Interactive Concepts
- Towards a Unified Game-Theoretic View of Adversarial Perturbations and Robustness
- Extracting Robust On-Manifold Interactions Encoded by Neural Networks
- I2MoE: Interpretable Multimodal Interaction-aware Mixture-of-Experts
- Mixture of Multimodal Interaction Experts
- Conditional Adversarial Domain Adaptation
- CMDA: Cross-Modal and Domain Adversarial Adaptation for LiDAR-Based 3D Object Detection
- Multi-Source Domain Adaptation with Mixture of Experts
- Sparse Mixture-of-Experts are Domain Generalizable Learners