Where Backimodal Backdoors Hide: Subspace Segregation in Disentangled Multimodal Models

Multimodal models — systems that learn from several input channels at once, such as text, audio, and video — can be built in two very different ways. The common way fuses everything into one joint representation. A less common way first splits each modality's features into two parts: a shared subspace holding the semantics that all channels agree on (the sentiment of a sentence, the identity of a face), and a private subspace holding what only that channel can express (acoustic pitch, visual texture). A backdoor attack, meanwhile, hides malicious behavior in a model during training: the model looks perfect on clean data, but flips to an attacker-chosen output whenever a trigger appears. This proposal asks a question nobody has asked: when a backdoor is planted in a model that keeps shared and private features separate, where does the backdoor go?

The question matters in both directions. For defenders, if the architectural separation forces the malicious circuit to settle in one subspace rather than contaminating both, then a defense can surgically retrain or reset only that subspace, at a fraction of the clean-data cost of full fine-tuning. For attackers, a trigger deliberately placed in a modality-private channel (a pitch shift in audio, a texture pattern in video) would leave the cross-modal semantics untouched — and every leading multimodal backdoor detector we know of reads exactly those cross-modal semantics, so it would be blind. Both halves follow from one hypothesis, which we state below and can falsify with existing benchmarks.

Background and the Gap

Two literatures set the stage. On the disentanglement side, MISA projects text, audio, and video features into modality-invariant and modality-specific subspaces, aligning the invariant parts with a moment-matching loss and keeping the specific parts apart with an orthogonality penalty; DecAlign extends this with prototype-based alignment for the unique streams. CH-SIMS shows why the separation helps: training only on a global consensus label corrupts the internal representations, and auxiliary unimodal labels preserve each channel's distinctiveness. Crucially, none of these architectures has ever been subjected to a backdoor attack — their evaluations stop at clean benchmarks. On the backdoor side, the picture inverts. BadCM crafts invisible triggers that live precisely in the modality-invariant components that bridge image and text, because perturbing the shared semantics is what moves the fused output — but it never models the private subspace. InverTune and BDetCLIP defend contrastive models whose encoders are monolithic, and both explicitly leave open what happens if shared and private features are architecturally separated.

Three results make the intersection more than idle curiosity. First, FST documents an "entanglement paradox" in unimodal vision: at low poisoning rates (0.5%–5%), backdoored features occupy the same latent region as clean features, so standard fine-tuning fails — the attack is dangerous precisely when it is entangled. Whether multimodal orthogonality constraints break this entanglement, force it into one subspace, or change nothing is unknown. Second, When One Modality Rules Them All shows that multimodal backdoors in a monolithic fusion model (InstructPix2Pix) collapse onto the text trigger almost entirely — text attribution $\phi_T = 0.9532$ versus image attribution $\phi_I = 0.0045$ — and the authors explicitly leave open whether architectural decoupling changes this. Third, BDetCLIP detects poisoned images through "semantic insensitivity": a triggered image matches its target class regardless of whether the text prompt is sensible or corrupted. This mechanism, like the attention-entropy detectors BYE and TCAP, reads the model's cross-modal behavior — and would say nothing about a trigger whose entire effect is confined to a private subspace.

Core Claim

The claim is a segregation hypothesis plus its two consequences. In a multimodal model with enforced shared/private separation (MISA- or DecAlign-style), a low-poisoning backdoor cannot remain diffusely entangled across both subspaces: the orthogonality and reconstruction constraints push the malicious association into one of them — usually the shared subspace for cross-modal triggers, as BadCM implicitly exploits, and the private subspace for triggers with no cross-modal meaning. If segregation holds, then (a) a defender can identify the compromised subspace and repair only it, preserving clean accuracy with a very small clean-data budget; and (b) an attacker can deliberately confine the trigger to the private subspace, and detection methods that compare modalities against each other will fail. The same instrument measures both: a subspace-level trigger attribution.

Concretely, we extend the Trigger Modality Attribution (TMA) of When One Modality Rules Them All from modalities to subspaces. For a fused model output $f(x)$ and the two subspaces of modality $m$ (shared $s_m$, private $p_m$), we define

$$\phi_{S} = \frac{1}{|{\subset} S|}\sum_{T \subseteq S} \big[ v(S \setminus T) - v(S) \big],$$

a Shapley-style attribution where $S$ ranges over subspace ablations (zeroing out $s_m$ or $p_m$ for each modality before fusion) and $v(\cdot)$ is the attack success rate on triggered inputs. A backdoor is segregated when one subspace carries nearly all of $\phi$; entangled when attribution is diffuse. This is the ablation-based measurement philosophy of When One Modality Rules Them All, applied at a finer granularity the fusion architecture makes available for the first time.

Research Program

The program has three parts, each producing a standalone finding.

First, the measurement study. We train MISA-style and DecAlign-style models on CMU-MOSI, CMU-MOSEI, CH-SIMS, and UR_FUNNY, and poison the training data at 0.5%, 1%, and 5% rates with three trigger families: a BadCM-style shared-semantic trigger (synonym substitution or invariant-region perturbation), a Meme-Trojan-style cross-modal pipeline trigger (following Meme Trojan, a visual artifact that leaks into the text stream), and a deliberately private trigger (a pitch shift in audio or a texture patch in video with no textual correlate). We then measure subspace attribution. The decisive comparison is against an ablated architecture: the identical dual-encoder model with the orthogonality loss removed, so shared and private features are free to mix. If segregation appears only with the orthogonality constraint, the constraint — not the extra capacity — is the cause. This ablation is the single most important control in the proposal, because a skeptic will otherwise attribute any effect to the two-encoder architecture having more parameters.

Second, the defense. Given a segregated backdoor, we purge only the compromised subspace: fine-tune (or reset to a clean checkpoint, as DAM resets backdoor-sensitive weights) the projection heads feeding that subspace while freezing the rest. The design borrows the subspace-scoped-editing idea from Beyond Sample-Level Forgetting, which shows that decoupled modal-specific and modal-consistent factors allow surgical removal of knowledge without collateral damage — but applies it to trigger removal rather than concept unlearning, a use its authors explicitly did not evaluate. The headline quantity is the clean-data budget: FST needs 50 clean samples and 10–15 epochs for unimodal CIFAR-10; we test whether subspace-scoped repair matches that efficiency in multimodal fusion models where whole-model fine-tuning is known to be unreliable at low poisoning (FST).

Third, the adaptive attack. We construct a best-case private-subspace trigger and evaluate the leading detection principles against it. We port BDetCLIP's semantic-insensitivity test (BDetCLIP) to our setting: compare the model's response to coherent versus semantically corrupted text paired with the triggered audio/video. We also test an attention-based check in the spirit of BYE. The prediction is that both degrade toward chance, because the trigger never touches the cross-modal alignment. This is an honest red-team exercise: if either detector still catches the private trigger, that is a finding about the robustness of cross-modal detection we would report, and it would weaken the practical case for subspace-scoped defenses (though not the segregation finding itself).

A secondary question, folded into the measurement study, addresses When One Modality Rules Them All's open problem: we compute its Cross-Trigger Interaction (CTI) metric on disentangled architectures to see whether decoupling permits genuinely synergistic multi-key triggers (CTI > 0) that monolithic fusion suppresses, or whether gradient imbalance forces collapse into the text channel regardless of architecture.

Baselines

The naive reference point is the poisoned model with no defense: we report its attack success rate and clean accuracy, establishing that the attack works and that clean performance is unimpaired (the standard stealth condition in BadCM and Meme Trojan). The strongest existing competitor is post-training fine-tuning on a small clean set, which BackdoorMBTI found to be the most robust defense across modalities in its 17-attack benchmark — it is simple, architecture-agnostic, and hard to beat on principle. We run it with the identical clean-data budget (50, 200, and 500 clean samples) and the identical epoch count as our subspace-scoped repair, so any difference is attributable to where the tuning happens, not how much. Beating it means reducing attack success rate below its value in the 0.5%–5% poisoning regime while keeping clean accuracy within one point of the unpoisoned disentangled model. The second strong competitor is FST (FST), the strongest published low-poisoning purification: it reduced CIFAR-10 attack success rate from nearly 90% to below 1% while holding clean accuracy around 93%, using 50 clean samples in 10–15 epochs. FST operates on a discrete linear classifier head, and the MISA-style sentiment models we use do end in such heads, so we adapt FST's weight-deviation penalty to them directly; no reported result found on Lacuna for FST on any multimodal benchmark, so we establish its multimodal numbers ourselves under the same tuning budget as ours. The third baseline is detection rather than repair: the semantic-insensitivity check ported from BDetCLIP, which reported an average AUROC of 0.938 on multimodal attacks and 0.972 on traditional attacks on ImageNet-scale evaluation, at roughly 3.8 minutes per 50,000 samples; we run its detection principle on both our shared-subspace and private-subspace triggers, with subspace-scoped detection (using attribution $\phi_S$ on a small triggered-probe set) as our counterpart. Finally, the ablation that isolates our key ingredient is the dual-encoder architecture with the orthogonality loss removed: same parameters, same data, same poisoning, only the shared/private separation absent. Every defense and detection result must be reported on both variants; if the defense works equally well without orthogonality, the disentanglement story is wrong and the credit belongs to architecture alone.

Why not just…?

Why not just fine-tune the model on clean data? This is the most serious alternative, and BackdoorMBTI's evidence is on its side at higher poisoning rates. But FST shows fine-tuning fails exactly in the 0.5%–5% regime we target, because the backdoor features share the latent space with clean features and nothing forces them apart. Our proposal does not argue fine-tuning away — we run it as the primary baseline under matched budgets — it argues that fine-tuning the whole model wastes the budget on parameters that were never compromised. If matched-budget whole-model fine-tuning matches our subspace-scoped repair in our experiments, the architectural claim collapses to a measurement curiosity, and we would report it that way.

Why not just detect at test time with prompt-based checks like BDetCLIP? BDetCLIP is fast and needs no model surgery, with AUROC of 0.938–0.972 on CLIP. But its mechanism reads the image–text semantic alignment; a trigger confined to the private subspace of an audio or video channel has no textual correlate, so the semantic-insensitivity signal should vanish. Our adaptive-attack experiment tests exactly this, and BDetCLIP's own authors flag the limitation for attacks that avoid text-semantic hijacking. Honest caveat: detection and repair are different goals — BDetCLIP cannot remove a backdoor, and our method's detection claim is confined to settings where the attacker uses the private subspace; on CLIP-style open-vocabulary models, InverTune (InverTune, which cut BadCLIP's attack success rate from 98.36% to 0.49% with 3.07% average clean-accuracy loss) remains the method to beat, and we do not claim to replace it there.

Why not just filter the training data with attention profiling, like BYE or TCAP? BYE reaches detection F1 above 95% and drives attack success rate to 0.03% on DocVQA, and TCAP achieves 100% precision/recall on one BadNet setting — but both are offline data-cleansing pipelines that require logging attention across training, fitting mixture models, and retraining from scratch. They target MLLMs with spatially localized triggers; TCAP itself notes BYE fails on diffuse triggers like blended patterns, and a pitch-based audio trigger causes no spatial attention collapse at all. For our compact fusion models on sentiment benchmarks, the attention infrastructure they assume does not exist. Still, if a defender can afford full retraining on filtered data, that is a stronger guarantee than any post-hoc repair — our method targets the setting where the poisoned model is already deployed or retraining is too expensive.

Why not just use a standard fused architecture and add data augmentation or label noise? BackdoorMBTI found label noise lifts detection accuracy only from 62.8% to 68.2% on average — helpful, not decisive. More fundamentally, the monolithic architecture is where the entanglement paradox (FST) and backdoor modality collapse (When One Modality Rules Them All) occur: with no subspace structure, there is nothing to scope a repair to and no finer-grained attribution to measure. Our orthogonality-ablation control is precisely the experiment that decides whether this architectural choice earns its cost.

Proposed Experiments and Decisive Checks

All experiments use the sentiment and affect benchmarks with published numbers for the clean architectures we poison: CMU-MOSI (MISA reports 83.4% binary accuracy; DecAlign reports F1 of 85.82), CMU-MOSEI (DecAlign F1 of 86.07), CH-SIMS (DecAlign 88.24% three-class accuracy versus 80.17 for the strongest baseline; CH-SIMS also ships independent unimodal labels for text, audio, and video, which makes it the ideal testbed for private-subspace attribution since we can verify that a pitch trigger moves the audio-only predictor and nothing else), and UR_FUNNY (MISA reports 70.61% accuracy). We poison at 0.5%, 1%, and 5% — the regime FST identifies as the hard one — under the three trigger families described above, and we reuse the BackdoorMBTI toolkit's attack implementations and noise conditions (BackdoorMBTI) where they transfer, so that attack strength is standardized rather than home-grown. Metrics throughout: attack success rate, clean accuracy (Acc-2, F1, MAE as each benchmark convention dictates), subspace attribution $\phi_S$, Cross-Trigger Interaction for the collapse question, detection AUROC, and clean-sample budget for repair.

The proposal succeeds or fails on three checks. Segregation is confirmed if, at 0.5%–5% poisoning with the orthogonality loss active, a single subspace carries the clear majority of attribution for each trigger family, while the no-orthogonality ablation shows diffuse attribution — the multimodal analogue of FST's entanglement, now with a measurable signature. The defense is confirmed if subspace-scoped repair drives attack success rate below whole-model fine-tuning at matched budget (targeting the <1% that FST achieves unimodally) while clean accuracy stays within one point of the published clean numbers. The adaptive-attack finding is confirmed if the semantic-insensitivity detector, which reaches 0.938–0.972 AUROC in its original setting, falls toward chance on private-subspace triggers while our attribution-based detection remains high. Any of these failing is itself informative: diffuse attribution in disentangled models would extend FST's entanglement paradox across modalities and close off this defense route; robust cross-modal detection of private triggers would show the detectors are stronger than their stated assumptions. Either way, the field learns where multimodal backdoors actually live.