Adversarial Robustness for Unified Multi-Modal Encoders via Efficient Calibration

Chih-Ting Liao, Bin Ren, Guofeng Mei, Xu Zheng

Abstract

Recent unified multi-modal encoders align a wide range of modalities into a shared representation space, enabling diverse cross-modal tasks. Despite their impressive capabilities, the robustness of these models under adversarial perturbations remains underexplored, which is a critical concern for safety-sensitive applications. In this work, we present the first comprehensive study of adversarial vulnerability in unified multi-modal encoders. We find that even mild adversarial perturbations lead to substantial performance drops across all modalities. Non-visual inputs, such as audio and point clouds, are especially fragile, while visual inputs like images and videos also degrade significantly. To address this, we propose an efficient adversarial calibration framework that improves robustness across modalities without modifying pretrained encoders or semantic centers, ensuring compatibility with existing foundation models. Our method introduces modality-specific projection heads trained solely on adversarial examples, while keeping the backbone and embeddings frozen. We explore three training objectives: fixed-center cross-entropy, clean-to-adversarial L2 alignment, and clean-adversarial InfoNCE, and we introduce a regularization strategy to ensure modality-consistent alignment under attack. Experiments on six modalities and three Bind-style models show that our method improves adversarial robustness by up to 47.3 percent at epsilon = 4/255, while preserving or even improving clean zero-shot and retrieval performance with less than 1 percent trainable parameters.

Introduction

Modern foundation models, such as ImageBind and UniBind, aim to create a unified embedding space where disparate modalities—including images, audio, video, and thermal data—are mapped to a shared mathematical neighborhood. While these models enable powerful cross-modal tasks, they exhibit significant adversarial vulnerability. A minor, imperceptible perturbation to an input can cause a "near-collapse" of the representation, posing a critical safety risk for autonomous systems and medical AI.

Previous defense strategies often required computationally expensive full-model retraining and were primarily designed for image-text pairs. This work addresses the challenge of securing unified multi-modal encoders efficiently, particularly for fragile non-visual modalities like audio and point clouds, which often drop to near-zero accuracy under attack.

Methodology: The CALMARS Framework

The proposed solution, CALMARS (Clean-oriented Alignment and Latent Modeling + Multi-phase Adversarial Representation Stabilization), is a two-stage framework that enhances robustness while keeping the massive pre-trained encoders frozen. This "plug-and-play" approach utilizes lightweight projection heads, requiring less than 1% of the original model's parameters.

The CALMARS framework overview: Stage 1 (CALM) focuses on clean alignment and embedding calibration, while Stage 2 (MARS) stabilizes representations against adversarial inputs using geometric consistency and feature regularization.

Stage 1: CALM (Clean-oriented Alignment and Latent Modeling)

This stage aligns the projection head with the clean data distribution. It employs feature distillation to mimic the frozen encoder's behavior and "Embedding Calibration" to ensure class separation in the latent space, preventing semantic overlap that can occur during training.

Stage 2: MARS (Multi-phase Adversarial Representation Stabilization)

In this phase, the model is exposed to adversarial examples. The framework enforces Geometric Consistency, ensuring that perturbed inputs map to the same neighborhood as their clean counterparts. This stabilization is achieved through multi-phase objectives including feature regularization and robust calibration.

Additionally, the authors introduce CrossMaxim, a non-targeted "machine-gun sweep" attack. Unlike targeted attacks that push an input toward a specific wrong label, CrossMaxim repels representations away from their true matches in all directions, revealing deeper vulnerabilities.

Experiments and Results

The researchers evaluated CALMARS across eight datasets covering six modalities: Image (ImageNet-1K, Places-365), Video (MSR-VTT, UCF-101), Audio (ESC-50, AudioCaps, Clotho), Event Streams (N-ImageNet1K), Point Clouds (ModelNet40), and Thermal (LLVIP).

Performance and Robustness

A key finding is that adversarial robustness does not necessarily come at the cost of clean accuracy. In many cases, the calibration process actually improves performance on uncorrupted data. For instance, on ImageNet-1K, CALMARS increased clean accuracy from 74.7% to 95.3% while maintaining 32.1% robustness under attack (at epsilon = 8/255), whereas the original UniBind model collapsed to 0%.

Comprehensive classification results across eight datasets and six modalities under varying adversarial attack strengths. CALMARS consistently outperforms existing defenses like FARE and TeCoA in both clean accuracy and adversarial robustness.

Efficiency and Transferability

By training only the projection heads, CALMARS achieves state-of-the-art results with minimal computational overhead—fixing security flaws in a few GPU hours. Furthermore, the robustness gained is transferable across different Bind-style models (e.g., from ImageBind to LanguageBind), suggesting the creation of a more structurally stable feature space.

Discussion and Conclusion

CALMARS demonstrates that unified multi-modal encoders can be secured efficiently without sacrificing performance. However, limitations remain: the framework still requires modality-specific projection heads, and highly complex attacks like CrossMaxim still cause significant degradation. Future research should explore modality-agnostic architectures and real-time adaptive defense mechanisms to counter evolving adversarial threats in safety-critical multi-modal applications.

Summary updated

Abstract

Recent unified multi-modal encoders align a wide range of modalities into a shared representation space, enabling diverse cross-modal tasks. Despite their impressive capabilities, the robustness of these models under adversarial perturbations remains underexplored, which is a critical concern for safety-sensitive applications. In this work, we present the first comprehensive study of adversarial vulnerability in unified multi-modal encoders. We find that even mild adversarial perturbations lead to substantial performance drops across all modalities. Non-visual inputs, such as audio and point clouds, are especially fragile, while visual inputs like images and videos also degrade significantly. To address this, we propose an efficient adversarial calibration framework that improves robustness across modalities without modifying pretrained encoders or semantic centers, ensuring compatibility with existing foundation models. Our method introduces modality-specific projection heads trained solely on adversarial examples, while keeping the backbone and embeddings frozen. We explore three training objectives: fixed-center cross-entropy, clean-to-adversarial L2 alignment, and clean-adversarial InfoNCE, and we introduce a regularization strategy to ensure modality-consistent alignment under attack. Experiments on six modalities and three Bind-style models show that our method improves adversarial robustness by up to 47.3 percent at epsilon = 4/255, while preserving or even improving clean zero-shot and retrieval performance with less than 1 percent trainable parameters.
BibTeX
@misc{liao2025adversarial,
  author = {Chih-Ting Liao and Bin Ren and Guofeng Mei and Xu Zheng},
  title = {{Adversarial Robustness for Unified Multi-Modal Encoders via Efficient Calibration}},
  year = {2025},
  eprint = {2505.11895},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2505.11895},
}
Figure