Unified multimodal encoders map images, text, audio, point clouds and other signals into one shared embedding space. Examples are ImageBind and UniBind, and in the file they are called Bind-style encoders. They are very fragile. CALMARS reports that non-visual modalities such as audio and point clouds fall to near-zero accuracy under mild adversarial perturbations, meaning small crafted changes to the input that flip the prediction. The cheapest known repair freezes the large encoders and trains small projection heads on top, under 1% of the parameters. That repair still pays the expensive part of adversarial training. Every head update needs fresh adversarial examples, and each one is built by several rounds of gradient steps pushed back through the frozen multi-hundred-million-parameter backbone. The head is cheap, but finding the attack against it is not.
We propose to move that attack search out of the input and into the small feature space that the head reads. The search is also restricted to the few directions along which real input attacks can actually move those features. The head is then trained so that a perturbed audio (or point-cloud) embedding stays aligned with the clean embedding of its paired sample in a robust anchor modality such as text. If this works, the gradient needed to build each perturbation comes from the head's own backward pass at no extra cost, the backbone is never differentiated during training, and the cost of robustness drops to roughly that of ordinary head fine-tuning. The claim to test is that this cheap, cross-modally anchored training reaches a clean-versus-robust accuracy trade-off close to multi-step head training at a small fraction of the compute, when measured by attacks on the raw input of the full frozen pipeline.
Core Claim
Freeze the backbone and let $z = g(x)$ be its output feature for input $x$. Any input attack $x+\delta$ with $\|\delta\|_\infty \le \epsilon$ can only move $z$ within a set that is approximately $J\delta$, where $J$ is the backbone's input Jacobian. Empirically, backbone Jacobians concentrate most of their energy in a low-dimensional subspace. If so, a head trained against worst-case feature perturbations inside that subspace, with a magnitude matched to $\epsilon$, should inherit robustness to real input attacks.
We estimate the subspace once, offline, by power iteration on the average of $JJ^\top$ over a few thousand training inputs. This is a one-time cost of a few hundred backbone backward passes, and it is reused for all of training. The training objective for head $h$ on a fragile modality $m$, with paired anchor embedding $a$ from the robust modality, is
$$\min_h \; \mathbb{E}\Big[\, \ell\big(h(z), a\big) + \lambda \, \max_{\eta \in P_k(\rho)} \ell\big(h(z+\eta), a\big) \Big],$$
In this objective, $\ell$ is a contrastive alignment loss against the anchor and the other items in the batch, $P_k(\rho)$ is the set of perturbations lying in the top-$k$ Jacobian subspace with norm at most $\rho$ (where $\rho$ is calibrated from $\epsilon$ and the Jacobian singular values), and $\lambda$ sets the clean-versus-robust trade-off.
The inner maximum is solved with one signed gradient step with respect to $z$. That gradient is already computed when backpropagating through $h$, because $z$ is the head's input. The idea follows SLAT, which perturbs hidden layers with gradients obtained "for free" and shows this avoids the collapse single-step training usually suffers. We apply it where it is cheapest: at the boundary of a frozen giant model.
Relation to Prior Work
Three kinds of prior work shape this proposal.
The first motivates it and supplies the evaluation setting. CALMARS establishes that frozen-backbone heads can restore robustness in Bind-style encoders. It trains them with a clean-alignment stage and a stage that pulls perturbed inputs toward clean ones, but it keeps iterative input-space attack generation, and it aligns to fixed per-class semantic centers. We keep the frozen-head setup and change two things. The attack lives in a Jacobian-restricted feature subspace, and the anchor is the paired sample from the robust modality rather than a fixed center. That second change is where the adversarial modality alignment direction enters: the inner maximization is literally an attack on cross-modal alignment, and the head wins the min-max by restoring it.
The second supplies the method. TRADES splits robust error into clean error plus the probability of lying near the decision boundary, and uses a weight $\lambda$ to trade the two off. We adopt its structure but measure the boundary term against a cross-modal anchor. SLAT shows single-step latent perturbation runs 3.7x faster than seven-step projected gradient descent (PGD-7) without catastrophic overfitting. However, it was tested only on small convolutional networks with full end-to-end training, so whether it transfers to a frozen transformer backbone is the open question we test.
The third is prior work that bypasses training entirely and constrains our design. COLA projects CLIP image features onto a subspace spanned by class text embeddings at test time, and ATAC corrects embeddings using drift under augmentation. Both are vision-only, both add inference cost (COLA reports 28 minutes on ImageNet), and COLA needs the class names in advance. Robustness Tokens tune ten input tokens in 200 steps on a frozen vision transformer, keeping clean accuracy at 85.2% versus 85.6%. They rely on the tokens staying secret and have not been tested on non-visual modalities.
The collapse analysis in A Closer Look at Multimodal Representation Collapse explains why the weak modality needs its own treatment. Shared representations starve noisier modalities of capacity, which fits the finding that audio and point clouds break first. That work's own fix adds a modality discriminator and extra training complexity. We avoid a discriminator entirely, because pointwise anchoring is simpler and more directly tied to the attack.
Research Program
The work has three parts, each designed to fail visibly if the premise is wrong.
The first part tests the premise. For ImageBind and UniBind, and for each non-visual modality, we measure how much of the actual feature displacement caused by input-space PGD attacks against the full pipeline falls inside the top-$k$ Jacobian subspace, as $k$ varies. If the captured fraction is low at any small $k$ (for example under half at $k=64$), a fixed dataset-level subspace is the wrong object. In that case we fall back to refreshing a per-batch subspace with one real input gradient every $K$ steps, and we report the cost that adds.
The second part builds the training method: subspace estimation, single-step anchored latent perturbation, and a sweep of $\lambda$ that traces a clean-versus-robust curve for each modality. We count compute in backbone forward and backward passes, as well as GPU-hours, since backbone passes dominate the cost.
The third part is adversarial evaluation of the final pipeline on raw inputs only. We use PGD with many steps and restarts, the multi-directional sweep attack CrossMaxim introduced by CALMARS, and an adaptive attack that knows the head and the subspace and targets the directions outside it. The last attack is the honest test of whether we have merely hidden the gradient.
Baselines
The naive reference is the undefended frozen encoder with its original heads. It anchors the clean accuracy we must not lose, and it gives the near-zero robust accuracy CALMARS reports for non-visual modalities.
The strongest existing method is CALMARS, reproduced with its multi-step attack generation. It reports up to a 47.3-point robustness gain at $\epsilon=4/255$, and on ImageNet-1K clean accuracy rising from 74.7% to 95.3% with 32.1% robust accuracy at $\epsilon=8/255$, where undefended UniBind scores 0%. I found no reported CALMARS number on Lacuna broken out separately for the audio or point-cloud benchmarks, so we will reproduce those ourselves.
We also run an upper-reference variant that uses our cross-modal anchored loss with full multi-step input-space PGD. This separates the effect of the loss from the effect of the cheap attack.
All methods share the same head architecture, training data, and number of optimizer steps, and each gets the same hyperparameter search budget over $\lambda$ and learning rate. We beat the strongest baseline if, at matched clean accuracy (within one point), robust accuracy under the strongest adaptive attack is within three points of multi-step training at no more than one fifth of its backbone-pass budget. Alternatively, we beat it if our curve lies above CALMARS's clean-versus-robust curve at equal compute.
Two ablations isolate the key ingredients. The first replaces the Jacobian subspace with perturbations drawn in all feature directions (isotropic latent perturbations). The second replaces the paired cross-modal anchor with fixed class centers.
Why not just…?
Why not just generate adversarial features once, cache them, and train the head on the cache? This is the most serious threat, because it also avoids per-step backbone gradients. Its weakness is that cached perturbations were computed against the initial head. As the head changes, those perturbations stop being worst-case, which is the classic failure of static adversarial data. We run it at a matched one-time budget. If it matches our method under adaptive attack, then the subspace machinery is unnecessary, and we would report that the finding is "caching suffices for frozen heads."
Why not just use single-step input-space attacks (the fast gradient sign method, FGSM) with random initialization? This still needs one backbone backward pass per step. It also invites the catastrophic overfitting that SLAT documents, where the model learns to fool the single-step attack but collapses against multi-step PGD. We include it at equal wall-clock time, so the comparison directly shows whether our savings are real and whether its robustness survives many-step PGD.
Why not just defend at test time, as COLA does? COLA recovers robust accuracy from 1.0% to 77.2% averaged over nine CLIP datasets. However, it needs class names in advance, adds heavy per-query cost, and projects onto a text-spanned subspace that an attacker aware of the projection can target. We port its projection step to audio embeddings in ImageBind as a baseline. If it holds up against a projection-aware attack at acceptable latency, the training-based route loses its motivation for that modality.
Why not just use fewer PGD steps in CALMARS? This is cheap to try, so we sweep two-step and three-step versions. Even two steps still cost two backbone backward passes per update, whereas our method costs none during training. The compute-matched curves decide whether that difference matters.
Decisive Checks
The central experiment trains heads on the audio and point-cloud modalities of ImageBind and UniBind, plus the image modality on ImageNet-1K for comparability with the published CALMARS numbers. We then plot clean zero-shot accuracy against robust accuracy under adaptive PGD and CrossMaxim at $\epsilon \in \{4/255, 8/255\}$, with total backbone passes on the third axis.
The proposal succeeds if our curve sits near the multi-step anchored reference and clearly above both the cached-feature baseline and FGSM at equal compute. It fails, and we will say so, in either of two cases. The first is that the attack aimed at directions outside the subspace recovers most of the undefended vulnerability, which would mean latent robustness did not transfer to the input. The second is that the fixed-center ablation performs as well as the cross-modal anchor, which would mean the alignment component adds nothing.
References
- Adversarial Robustness for Unified Multi-Modal Encoders via Efficient Calibration
- Reliably fast adversarial training via latent adversarial perturbation
- Theoretically Principled Trade-off between Robustness and Accuracy
- Enhancing CLIP Robustness via Cross-Modality Alignment
- ATAC: Augmentation-Based Test-Time Adversarial Correction for CLIP
- Robustness Tokens: Towards Adversarial Robustness of Transformers
- A Closer Look at Multimodal Representation Collapse