Late-Added Senses: Does a Plug-In Adapter Make Multimodal Fusion Insensitive to How Long the First Modality Was Trained?

Human experts show two characteristic failures when a new sense is added to a skill they learned through one sense. Sometimes they ignore the new cue. Experienced sleep technicians gained nothing from hearing EEG signals turned into sound, while novices did (EEG Sonification improves sleep staging performance in novice stagers). Other times they cannot stop processing a cue that is irrelevant. Pianists' sound judgments got worse when irrelevant touch was present, and nonmusicians' did not (Multisensory interactions on auditory and somatosensory information in expert pianists). Musicians also recalled visual digits worse when the digits were paired with rhythmic tones (With No Attention Specifically Directed to It, Rhythmic Sound Does Not Automatically Facilitate Visual Task Performance). Neural networks have a close analogue. When an encoder for one modality is pre-trained alone and then fused with a second, the length of that solo pre-training controls the result. Changing it moved CREMA-D emotion-recognition accuracy from 60.1% to 81.3% (Shaping Initial State Prevents Modality Competition in Multi-modal Fusion). That paper's fix is to time the switch precisely, at the epoch where information the two modalities carry only jointly peaks.

This note asks whether a structural change makes that timing unnecessary. The change is to freeze the mature encoder and attach the new modality through a small cross-attention adapter that starts switched off. A cross-attention adapter is a side module that lets the frozen network read the new modality's features without its own weights changing. The robotics system DECO used this design to add touch to a trained vision policy (DECO: Decoupled Multimodal Diffusion Transformer for Bimanual Dexterous Manipulation with a Plugin Tactile Adapter), but only at a single backbone maturity. We propose to map final fused accuracy against how long the first encoder was trained, for modular addition and for joint fine-tuning side by side. We also test whether the adapted system shows either expert failure mode: ignoring a useful new cue, or being hurt by a useless one.

Core Claim

Call $T$ the number of epochs the first ("incumbent") encoder trains alone before the second modality is added. Call $A(T)$ the final fused test accuracy after a fixed budget of fusion training. For joint fine-tuning, Tang et al. report that $A(T)$ rises and then falls. We claim that a frozen incumbent plus a zero-initialized, gated cross-attention adapter gives an $A(T)$ curve that is flat or rising in $T$. The adapter should also match or beat the best point on the joint-fine-tuning curve, without a controller that predicts that point.

There are two possible reasons the joint curve falls, and the experiments are designed to tell them apart. If the decline is an optimization effect, where the mature modality dominates shared gradients, freezing should remove it. If it is a representational effect, where over-specialized features have discarded information the other modality could have combined with, freezing will not help. In that case only an adapter that reads earlier layers should recover. Either outcome tells practitioners whether "add a sense to a mature model" is a scheduling problem or a representation problem.

Research Program

The prior work falls into three groups by role. The first group supplies the motivation. Modality Competition: What Makes Joint Training of Multi-modal Network Fail in Deep Learning? (Provably) proves that joint late-fusion training locks onto whichever modality learns fastest and leaves the other encoder near initialization. Tang et al. show the outcome is set by encoder state at the moment fusion starts. Both study joint training only.

The second group supplies the method. DECO's two-stage freeze-then-adapter recipe updates 7.97M of 91M parameters and raises average success from 72.25% to 82.50% on DECO-50. Its ablation shows that which pathway a modality enters through matters: moving touch into the layer-normalization pathway collapsed performance. MultiMoDN—Multimodal, Multi-Task, Interpretable Modular Networks shows that sequential modular updates to a shared state can match parallel fusion. It explicitly leaves open whether a highly trained early module biases that state.

The third group constrains the design and supplies the strongest competitors. These are balanced joint-training methods that avoid competition from scratch: alternating per-modality updates in ReconBoost: Boosting Can Achieve Modality Reconcilement, Shapley-weighted gradient scaling in Boosting Multi-modal Model Performance with Adaptive Gradient Modulation, and conflict-aware gradient combination in MMPareto: Boosting Multimodal Learning with Innocent Unimodal Assistance. None of these addresses the late-addition setting, where an expert model already exists and should not be retrained.

We will build a single codebase with standardized interfaces for encoders, fusion modules and training schedules. This follows the modular-pipeline argument of DeepDIVA. Every method then shares data loading, backbones, augmentation and metrics, and only the fusion and schedule differ.

The adapter inserts cross-attention blocks from the incumbent's layers to the new modality's encoder. Each block's output is multiplied by a learned scalar gate that starts at zero, so at step zero the system is exactly the expert unimodal model. A second variant makes the gate depend on the input, which lets it suppress the new modality case by case. That variant is the computational counterpart of the selective gating that pianists appear to lack.

For each benchmark we checkpoint the incumbent at a sweep of $T$ values, from under-trained to well past convergence. Each checkpoint is fused under every method with an identical fusion budget. Throughout, we track Tang et al.'s fast information-decomposition estimates of unique, redundant and synergistic information. This tells us whether the adapter actually extracts synergy, meaning information present only in the combination of modalities, or merely averages two opinions. We run both directions of addition: the slow visual modality added to a mature audio model on CREMA-D, and the reverse. The pianist data show a strong asymmetry, with irrelevant touch hurting sound judgments but not the converse, and we test whether a network trained to expertise in one modality shows a matching directional effect.

Baselines

The do-nothing reference is the incumbent unimodal model at each $T$, alongside naive concatenation fusion trained jointly from scratch. On CREMA-D, concatenation fusion is known to starve the visual encoder: its standalone visual accuracy is 26.81%, against 73.01% under ReconBoost (ReconBoost). The adapter must clear the unimodal model at every $T$. Clearing it only at small $T$ would itself be a blocking effect.

The most direct competitor on this question is joint fine-tuning of both encoders after Stage I, with and without Tang et al.'s synergy-peak controller. That controller is the existing answer to the maturity problem, reaching a best of 81.3% on CREMA-D and an average gain of 7.70% across CREMA-D, AVE, UCF-101 and CGMNIST (Shaping Initial State). We re-run it on our shared backbones at every $T$. Beating it means two things: equal or higher accuracy than its controller-chosen transition, and a range of $A(T)$ across the sweep that is less than a third of its range.

The strongest from-scratch balanced methods are ReconBoost, reported at 79.82% on CREMA-D and 71.35% on AVE, and MMPareto, reported at 75.13% on CREMA-D against 71.10% for uniform loss averaging. AGM is reported as 17.34 points above standard late-fusion joint training on CREMA-D. These methods answer a slightly different question, since they retrain everything. For every method, total epochs (Stage I plus fusion) are matched, and all methods get the same number of hyperparameter trials. The published numbers serve only as sanity anchors, because their backbones may differ from ours.

Our adapter does not need to beat them outright. It needs to come within about one point of them while training under 10% of the parameters during fusion and leaving the expert model unchanged.

Three ablations isolate the key ingredient. The first unfreezes the backbone under the same adapter, which tests whether freezing is what removes the maturity effect. The second injects the new modality by concatenation instead of gated cross-attention, which tests DECO's pathway finding outside robotics. The third restricts the adapter to read only the incumbent's final layer, which separates the optimization explanation from the representational one.

Why not just…?

Why not just ensemble two independently trained unimodal models? This is the most serious threat. Averaging the outputs of separately trained encoders has no gradient competition and no dependence on $T$, so it trivially achieves a flat curve. MMPareto's data hint that isolated training can leave each encoder fairly strong: standalone audio reached 61.69% on CREMA-D. The adapter is only worth having if it captures interactions an ensemble cannot. We therefore report the ensemble at every $T$ and compare synergy estimates from the information decomposition. We give extra weight to CGMNIST, where the deliberate bias rewards combining modalities over shortcuts. If the adapter does not beat the ensemble on CGMNIST and CREMA-D, we will report that the late-addition problem reduces to ensembling. The claim would then shrink to a statement about when fusion is worth the trouble.

Why not just use the synergy-peak controller? It works, but it assumes you control Stage I. The setting that motivates this note is an existing expert model, such as a deployed vision policy or a clinical classifier, that was trained to convergence by someone else. In that setting the peak may already have passed. The $T$-sweep measures directly how much accuracy the controller loses when started late versus how much the adapter loses.

Why not just retrain jointly with ReconBoost or MMPareto? Retraining discards the expert model and costs a full training run. It also risks the forgetting of learned skills that DECO cites as its reason for freezing. The matched-budget comparison above prices this trade-off. If balanced retraining is dramatically better at equal compute, the case for modular addition is weak, and we will say so.

Proposed Experiments and Decisive Checks

The main experiments run on CREMA-D, AVE, Kinetics-Sounds and CGMNIST, the benchmarks shared by the competing methods. A transfer check runs on DECO-50 (BAAI-Humanoid/DECO-50). There we vary how long the vision-action policy trains before the tactile adapter is attached, and measure average success, focusing on the occlusion-heavy Assembly and Waste Disposal tasks.

The first decisive check is the shape of $A(T)$. A flat adapter curve beside a peaked joint curve supports the core claim. A declining adapter curve that recovers only with multi-depth reading supports the representational explanation.

The second check measures blocking. We record the fraction of synergy captured by the adapter, and the adapter's gate magnitude, as $T$ grows. A gate that stays near zero at large $T$, with no accuracy gain over the unimodal model, is the network analogue of the expert sleep stagers.

The third check measures interference. At test time we replace the added modality with mismatched samples from other clips, or with noise. We then compare the accuracy drop below the unimodal incumbent across fixed-gate, input-gated and jointly fine-tuned models, in both directions of addition.

The expected result is that joint fine-tuning at large $T$ shows the pianist pattern, gaining from congruent inputs and losing from irrelevant ones. The input-gated adapter should keep the gain while dropping no more than one point below the unimodal model under mismatch. If the gated adapter shows the same interference as joint training, then input-dependent gating does not provide the selective suppression the human data suggest is missing. That result would argue against claims that modular routing alone confers it.