When a model that hears and sees is trained to resist attacks on one of its inputs, it has two ways to get there. It can learn to read the attacked input more carefully, or it can learn to stop reading it and lean on the other input instead. The second route is cheaper. It produces good numbers against the attack used in training, but it leaves a model that collapses once the attacker switches to the input it now trusts. This proposal claims that the cheap route is the default behavior of parameter-efficient adversarial training on multimodal models. Parameter-efficient training freezes a large pretrained network and updates only small add-on modules, such as low-rank adapters (small matrix pairs added to existing weights). The claim is that under this setup, adversarial pressure on one modality speeds up modality dominance: the familiar failure in which a multimodal network relies on its easiest input and starves the others.
We propose to measure this effect on audio-visual benchmarks, show whether parameter restriction makes it worse than full fine-tuning, and fix it with a small change to the training loop. The adversary chooses how to split a single perturbation budget across modalities. Each modality's adapter has its update scaled by how robust that modality is on its own under attack, not by how confident it is on clean data. Robustness is then judged by the worst case over modalities, including attacks that switch modality after training. The question both literatures have left open is whether a modality "becomes robust" or just "becomes ignored."
Where the two literatures stand
The parameter-efficient defense work is almost entirely single-modality in its threat model. AdvLoRA argues that adversarial adaptation of vision-language models is intrinsically low-rank and constrains it to adapter matrices. AdvCLIP-LoRA runs few-shot minimax training on CLIP adapters and finds that the attention value projections are the most important place to put them. AdvMask learns binary masks over frozen attention weights. DEFEAT snaps visual patch features onto a discrete codebook. All four perturb only the image and treat the text side as a fixed, trusted anchor. That design choice is exactly what would let a model "solve" the visual attack by drifting toward the unattacked channel. Their reported robustness therefore cannot tell repair apart from abandonment. These papers supply our adapter machinery: low-rank adapters placed on value projections following AdvCLIP-LoRA. They also motivate the question.
The modality-dominance work supplies the diagnosis and the modulation tools, but it assumes clean inputs. Wang et al. show that joint audio-visual training on Kinetics can underperform RGB alone (71.7% versus 72.6%) because modalities generalize at different rates. OGM-GE scales down the gradients of the more confident modality branch and adds noise back. MMPareto shows that off-the-shelf multi-objective balancing performs worse than plain averaging, because it suppresses useful gradient noise; this constrains our design. PMR frames dominance as directional interference with the weak modality's class clusters. None of these models an adversary. The confidence signal they modulate on is computed on clean data, which is the wrong signal once one modality is under attack.
Three bridging papers come closest, and each leaves the central question open. VARMAT is the nearest neighbor. It computes a first-order per-modality vulnerability score (feature magnitude times gradient norm) and penalizes the gradient norm of the most vulnerable modality during full-parameter adversarial training on MultiBench. It treats vulnerability as a static sensitivity. It does not ask whether training shifts reliance between modalities, and it does not work in adapter subspaces. MA-AVT adapts frozen audio-visual ViTs with learned tokens and blockwise contrastive alignment, but only on clean data. FlowGuard shows that attacks on multimodal LLMs fracture cross-modal information flow, but it only detects attacks at inference and trains nothing. Our contribution differs from all three in its object of study. We track how per-modality reliance moves during adversarial adapter training, and we evaluate under an adversary that can pick its modality.
Core Claim
Adversarial training of small adapters on a frozen audio-visual backbone, against perturbations in one modality, mainly buys robustness by lowering the model's reliance on that modality. We call this robustness by abandonment.
We predict the effect is stronger under parameter restriction than under full fine-tuning at matched robust accuracy. The reason comes from the tension between the two literatures. Low-rank constraints damp gradient variance to stop the model from memorizing adversarial noise, while dominance work says that variance is what lets the weaker branch recover. A constrained subspace makes "re-weight the fusion toward the clean modality" much easier to reach than "rebuild robust features in the attacked encoder."
We further claim that two changes remove most of this effect without reducing clean accuracy:
- an adversary that splits a joint budget across modalities;
- modulation of each modality's adapter by its robust standalone confidence.
Research Program
The first stage is diagnosis. We take a frozen ViT-B used for both spectrograms and frames, in the MA-AVT style. We attach separate low-rank adapters to each modality's encoder plus a small fusion head. We train three variants: attack audio only, attack video only, and attack both with fixed per-modality budgets.
Throughout training we log three quantities:
- Standalone strength. The clean and adversarial accuracy of a linear probe on each modality's encoder output.
- Reliance. The accuracy drop when one modality's input is replaced by its dataset mean.
- Switched-attack accuracy. Accuracy against an attack on the modality the model was not trained against.
Abandonment has a specific signature. After audio-only adversarial training, audio reliance falls, the audio probe's adversarial accuracy barely rises, and switched-attack accuracy on video ends up lower than for an undefended model. We repeat the diagnosis with full fine-tuning of the same backbone and compare at matched in-distribution robust accuracy. This isolates whether parameter restriction is responsible.
The second stage is the method. The adversary maximizes the loss under a single normalized budget shared across audio $a$ and video $v$:
$$\max_{\delta_a,\delta_v}\ \mathcal{L}\big(f(x_a+\delta_a,\ x_v+\delta_v),\,y\big)\quad \text{s.t.}\quad \frac{\|\delta_a\|}{\epsilon_a}+\frac{\|\delta_v\|}{\epsilon_v}\le 1.$$
Here $f$ is the multimodal model and $\epsilon_a,\epsilon_v$ are per-modality reference budgets. We set the split once per step from a first-order estimate, which is VARMAT's probe used as an allocator rather than a penalty: to first order, the loss gain from each modality is proportional to its budget times its gradient norm. Projected gradient steps (PGD) then run inside that split, so the cost per step matches attacking a single modality.
On the update side, let $s_m$ be the average softmax probability of the correct class from modality $m$'s standalone head on adversarial inputs. Let $\bar m$ denote the other modality. We scale modality $m$'s adapter gradients by
$$k_m = 1-\tanh\!\big(\alpha\cdot\max(0,\ s_m/s_{\bar m}-1)\big),$$
which is OGM-GE's functional form with its clean-confidence ratio replaced by a robust one; $\alpha$ is a single tuned scalar. We also add per-modality robust classification losses, summed with the fusion loss. These are combined by MMPareto's sign-of-cosine rule rather than standard Pareto weighting, since MMPareto shows the latter backfires. Nothing is added at inference: the deployed model is an ordinary network with adapters, so standard white-box attacks apply without gradient-masking concerns.
The third stage asks whether the effect is a general property of adapter defenses. We repeat the diagnosis with MA-AVT-style token adapters in place of low-rank ones. We also repeat it on AVMNIST, which allows a direct comparison with VARMAT's reported setting.
Baselines
Clean reference points. The naive reference is the frozen backbone with clean-trained adapters, attacked without any defense. It gives the floor for robust accuracy and the ceiling for clean accuracy. For clean accuracy on CREMA-D, the strongest reported numbers in the file are 75.13% for MMPareto (MMPareto, a different backbone) and 72.3% for the parameter-efficient MA-AVT (MA-AVT). On VGGSound, MA-AVT reports 56.7%. We expect our defense to cost some clean accuracy. We count it a success if clean accuracy stays within a few points of our own reproduced clean-trained adapter, since cross-backbone numbers are not directly comparable.
Strongest existing method. This is VARMAT (Vulnerability-Aware Robust Multimodal Adversarial Training), run two ways: as published with full parameters, and ported into the same adapters with its gradient-norm penalty on the vulnerable modality. It is the only method in the file that explicitly targets cross-modal vulnerability asymmetry during adversarial training. The file reports its absolute robustness gains of 11.19% on AVMNIST, 12.73% on CMU-MOSEI and 22.21% on UR-FUNNY. Absolute robust accuracies are not given, and there is no reported result found on Lacuna for adversarial accuracy of any method on CREMA-D, VGGSound or Kinetics-Sounds. We will therefore reproduce VARMAT on AVMNIST to anchor against its reported gain before porting it.
Adapted single-modality defense. This baseline is AdvCLIP-LoRA-style adversarial adapter training extended to attack both modalities with fixed equal budgets (AdvCLIP-LoRA). It represents what a practitioner would do by default.
Ablations. Two ablations isolate the key ingredients. The first keeps the joint-budget adversary but removes the robust-confidence modulation and unimodal losses, which tests whether a stronger adversary alone suffices. The second keeps the modulation but computes $s_m$ on clean inputs, which reduces to OGM-GE on top of adversarial training and tests whether the robust signal matters.
Fairness and success criterion. All methods share the backbone, adapter rank and placement, attack steps per update, total forward and backward passes, and an equal hyperparameter search budget. Beating the field means a higher worst-modality robust accuracy than every baseline at comparable clean accuracy. Worst-modality accuracy is the minimum over audio-only, video-only and joint-budget attacks.
Why not just…?
Why not just use modality dropout during adversarial adapter training? Dropout (Wolf et al.) is cheap and directly penalizes over-reliance, so it is a real threat. Our concern is that dropping the attacked modality teaches the model it can do without that modality, which could reinforce abandonment rather than repair. Wang et al. also report that generic regularizers do not fix differing generalization rates. We will run dropout and dropout plus our adversary under the same budget. If dropout matches our method on worst-modality and switched-attack accuracy, we will report it as the simpler answer and drop the modulation claim.
Why not just attack every modality uniformly? Fixed per-modality budgets do not model an attacker who concentrates on the weakest link. VARMAT argues that modality-blind perturbation both wastes budget on robust channels and over-perturbs fragile ones. The adapted AdvCLIP-LoRA baseline tests this directly under the joint-budget and switched evaluations.
Why not just stack OGM-GE on adversarial training? OGM-GE modulates on clean confidence. An attacked modality can be confident on clean data and still fragile, so the clean signal may throttle the wrong branch. The clean-signal ablation settles this.
Why not harden each encoder separately and fuse? FlowGuard shows that attacks can split intent across modalities so that each input looks benign alone. Frozen fusion over independently hardened encoders also inherits the late-fusion distortion described by PMR. We include this variant and evaluate it under the joint-budget attack.
Proposed Experiments
The benchmarks are CREMA-D, Kinetics-Sounds and VGGSound for audio-visual classification, with AVMNIST as the bridge to VARMAT. Attacks are multi-restart PGD on log-mel spectrograms and on frames, plus the joint-budget attack. Metrics are clean accuracy, per-modality and worst-modality robust accuracy, switched-attack accuracy, reliance drop under mean-replacement, and standalone probe robustness.
Three checks decide the proposal:
- Existence. Single-modality adversarial training must lower reliance on the attacked modality, and it must push switched-attack accuracy below the undefended model's, on at least two of the three audio-visual datasets. If not, abandonment is not real and the project reduces to an evaluation protocol.
- Cause. At matched robust accuracy, adapters must abandon more than full fine-tuning does. This tests whether parameter restriction is to blame.
- Fix. Our method must raise worst-modality robust accuracy over the ported VARMAT, dropout and clean-signal OGM ablations. The gain should come with higher standalone probe robustness in the attacked modality, which separates real repair from simply redistributing reliance.
References
- AdvLoRA: Adversarial Low-Rank Adaptation of Vision-Language Models
- Few-Shot Adversarial Low-Rank Fine-Tuning of Vision-Language Models
- Identifying Robust Neural Pathways: Few-Shot Adversarial Mask Tuning for Vision-Language Models
- Discrete Latent Features Ablate Adversarial Attack: A Robust Prompt Tuning Framework for VLMs
- What Makes Training Multi-Modal Classification Networks Hard?
- Balanced Multimodal Learning via On-the-fly Gradient Modulation
- MMPareto: Boosting Multimodal Learning with Innocent Unimodal Assistance
- PMR: Prototypical Modal Rebalance for Multimodal Learning
- Vulnerability-Aware Robust Multimodal Adversarial Training
- MA-AVT: Modality Alignment for Parameter-Efficient Audio-Visual Transformers
- Securing Multimodal AI through Internal Information Decomposition
- ModDrop: adaptive multi-modal gesture recognition