TL;DR
This paper introduces Feature-Space Smoothing (FS), a novel defense framework that provides mathematically provable (certified) robustness at the feature representation level rather than the final output. By pairing this framework with a plug-and-play Gaussian Smoothness Booster (GSB), the authors achieve strong protection for large models like Multimodal Large Language Models (MLLMs) against adversarial attacks without requiring expensive model retraining or alignment.
What Problem Does This Paper Address?
Deep learning models, including state-of-the-art MLLMs, are notoriously vulnerable to adversarial attacks—subtle, engineered perturbations to inputs that cause catastrophic failures. Traditional defenses like adversarial training are computationally prohibitive for massive MLLMs and often fail to generalize across different modalities.
On the other hand, certified defenses like Randomized Smoothing (RS) offer mathematical guarantees of robustness but are traditionally "output-centric" (e.g., limited to simple classification tasks). They do not easily translate to the open-ended, autoregressive generation found in MLLMs and incur massive computational overhead because they require thousands of forward passes through the entire model. This paper tackles the challenge of creating a generalized, computationally efficient certified defense that protects the underlying feature representations directly.
Methods
The authors propose Feature-Space Smoothing (FS). Instead of smoothing the final predictions, FS applies Gaussian smoothing directly to the feature encoder. This establishes a Feature Cosine Similarity Bound (FCSB), guaranteeing a lower bound on the cosine similarity between clean and adversarial feature representations under $\ell_2$-bounded perturbations.
To boost this bound without touching the frozen backbone encoder, they introduce the Gaussian Smoothness Booster (GSB). GSB consists of two lightweight, plug-and-play components:
- A Gaussian Denoiser placed before the encoder to mitigate input-level noise.
- A Residual Smoothness Mapper placed after the encoder to refine features and preserve the original feature distribution using a specialized statistical loss.
Furthermore, the authors demonstrate how this feature-level certification can analytically propagate to downstream prediction tasks (like top-1 retrieval or prototype classification) without modifying the prediction heads.
Datasets
The framework was evaluated across several standard benchmarks:
- ImageNet: Used for training the GSB module (on a subset of 50 classes) and evaluating certified/empirical image classification.
- MS COCO: Used to evaluate zero-shot top-1 and top-5 image-to-text retrieval robustness.
- NIPS 2017 Adversarial Attacks and Defenses Competition dataset: Used for evaluating image captioning performance under adversarial conditions.
- ScienceQA: Used for evaluating Visual Question Answering (VQA) task robustness.
Key Insights and Ideas
- Shift from Output to Feature Space: By certifying the feature space rather than the final output, the defense becomes task-agnostic. A robust feature representation naturally protects diverse downstream tasks (classification, retrieval, generation) simultaneously.
- The Power of Plug-and-Play: The GSB module allows researchers to protect massive, frozen pre-trained models (like LLaVA or OpenFlamingo) without spending millions on retraining or breaking cross-modal alignment.
- Geometry Matters: Prediction-wise certification depends not just on the encoder's robustness, but on the decision margin of the downstream task. For instance, image-text retrieval on COCO is harder to certify than ImageNet classification because candidate text embeddings are dense and semantically overlapping, leading to tighter decision boundaries.
Key Results
- Certified Radius Boost: For a noise level of $\sigma = 0.5$, adding GSB increased the average certified radius from 0.101 to 0.455 for CLIP, and from 0.134 to 0.514 for SigLIP 2.
- Empirical Protection: Under strong white-box attacks (like FOA), integrating FS into a vanilla LLaVA-1.5-7B model boosted image classification accuracy from a dismal 3.8% to 87.2%, while slashing the Attack Success Rate (ASR) from 81% to 0.2%.
- Efficiency: Unlike standard randomized smoothing which requires thousands of full-model forward passes, feature-space smoothing only requires smoothing the lightweight encoder. Setting the sample size $n_0 = 8$ increased LLaVA's per-image inference time by a mere ~0.36 seconds, making it highly practical for real-world deployment.
Limitations and Future Work
- Perturbation Type: Currently, the theoretical certification is restricted to $\ell_2$-bounded perturbations. Extending this mathematical framework to $\ell_\infty$ bounds or semantic/patch attacks remains an open challenge.
- Decoding Constraints: While the feature-space bound is general, translating it into a prediction-level guarantee currently relies on cosine-similarity-based decision rules (like retrieval or prototype classification). It does not yet provide formal prediction-level bounds for open-ended, autoregressive text generation.
Summary updated