Feature-Space Smoothing: Certified Robustness of Deep Representations

Song Xia, Meiwen Ding, Chenqi Kong, Wenhan Yang, Xudong Jiang

TL;DR

This paper introduces Feature-Space Smoothing (FS), a novel defense framework that provides mathematically provable (certified) robustness at the feature representation level rather than the final output. By pairing this framework with a plug-and-play Gaussian Smoothness Booster (GSB), the authors achieve strong protection for large models like Multimodal Large Language Models (MLLMs) against adversarial attacks without requiring expensive model retraining or alignment.

What Problem Does This Paper Address?

Deep learning models, including state-of-the-art MLLMs, are notoriously vulnerable to adversarial attacks—subtle, engineered perturbations to inputs that cause catastrophic failures. Traditional defenses like adversarial training are computationally prohibitive for massive MLLMs and often fail to generalize across different modalities.

On the other hand, certified defenses like Randomized Smoothing (RS) offer mathematical guarantees of robustness but are traditionally "output-centric" (e.g., limited to simple classification tasks). They do not easily translate to the open-ended, autoregressive generation found in MLLMs and incur massive computational overhead because they require thousands of forward passes through the entire model. This paper tackles the challenge of creating a generalized, computationally efficient certified defense that protects the underlying feature representations directly.

Methods

The authors propose Feature-Space Smoothing (FS). Instead of smoothing the final predictions, FS applies Gaussian smoothing directly to the feature encoder. This establishes a Feature Cosine Similarity Bound (FCSB), guaranteeing a lower bound on the cosine similarity between clean and adversarial feature representations under $\ell_2$-bounded perturbations.

Fig. 1. Illustration of the FS framework with GSB, which guarantees that the cosine similarity of the adversarial and clean features extracted by the MLLM’s encoder is larger than FCSB for robust predictions.

To boost this bound without touching the frozen backbone encoder, they introduce the Gaussian Smoothness Booster (GSB). GSB consists of two lightweight, plug-and-play components:

  1. A Gaussian Denoiser placed before the encoder to mitigate input-level noise.
  2. A Residual Smoothness Mapper placed after the encoder to refine features and preserve the original feature distribution using a specialized statistical loss.

Furthermore, the authors demonstrate how this feature-level certification can analytically propagate to downstream prediction tasks (like top-1 retrieval or prototype classification) without modifying the prediction heads.

Datasets

The framework was evaluated across several standard benchmarks:

  • ImageNet: Used for training the GSB module (on a subset of 50 classes) and evaluating certified/empirical image classification.
  • MS COCO: Used to evaluate zero-shot top-1 and top-5 image-to-text retrieval robustness.
  • NIPS 2017 Adversarial Attacks and Defenses Competition dataset: Used for evaluating image captioning performance under adversarial conditions.
  • ScienceQA: Used for evaluating Visual Question Answering (VQA) task robustness.

Key Insights and Ideas

  • Shift from Output to Feature Space: By certifying the feature space rather than the final output, the defense becomes task-agnostic. A robust feature representation naturally protects diverse downstream tasks (classification, retrieval, generation) simultaneously.
  • The Power of Plug-and-Play: The GSB module allows researchers to protect massive, frozen pre-trained models (like LLaVA or OpenFlamingo) without spending millions on retraining or breaking cross-modal alignment.
  • Geometry Matters: Prediction-wise certification depends not just on the encoder's robustness, but on the decision margin of the downstream task. For instance, image-text retrieval on COCO is harder to certify than ImageNet classification because candidate text embeddings are dense and semantically overlapping, leading to tighter decision boundaries.

Key Results

  • Certified Radius Boost: For a noise level of $\sigma = 0.5$, adding GSB increased the average certified radius from 0.101 to 0.455 for CLIP, and from 0.134 to 0.514 for SigLIP 2.
  • Empirical Protection: Under strong white-box attacks (like FOA), integrating FS into a vanilla LLaVA-1.5-7B model boosted image classification accuracy from a dismal 3.8% to 87.2%, while slashing the Attack Success Rate (ASR) from 81% to 0.2%.
  • Efficiency: Unlike standard randomized smoothing which requires thousands of full-model forward passes, feature-space smoothing only requires smoothing the lightweight encoder. Setting the sample size $n_0 = 8$ increased LLaVA's per-image inference time by a mere ~0.36 seconds, making it highly practical for real-world deployment.

Limitations and Future Work

  • Perturbation Type: Currently, the theoretical certification is restricted to $\ell_2$-bounded perturbations. Extending this mathematical framework to $\ell_\infty$ bounds or semantic/patch attacks remains an open challenge.
  • Decoding Constraints: While the feature-space bound is general, translating it into a prediction-level guarantee currently relies on cosine-similarity-based decision rules (like retrieval or prototype classification). It does not yet provide formal prediction-level bounds for open-ended, autoregressive text generation.

Summary updated

Abstract

Modern deep learning models exhibit strong capabilities across diverse applications, yet remain vulnerable to malicious inputs that induce erroneous predictions via feature-space distortion. To address this vulnerability, we propose Feature-space Smoothing (FS), a general defense framework that provides certified robustness at the feature representation level. We show that FS converts a given feature encoder into a smoothed variant that is guaranteed to maintain a certified lower bound on the cosine similarity between clean and adversarial features under l_2-bounded perturbations. We then establish that this Feature Cosine Similarity Bound (FCSB) can be extended to the prediction-wise certification under the cosine similarity measure, and the value of FCSB is determined by the encoder intrinsic Gaussian robustness score. Building on those insights, we introduce the Gaussian Smoothness Booster (GSB), a plug-and-play module to improve the encoder Gaussian robustness score. Specifically, the GSB module is plugged to enhance the feature-space consistency and maintain the feature utility for downstream tasks under Gaussian perturbations. This design enables seamless integration of FS on the protected model, e.g., Multimodal Large Language Models (MLLMs), without additional model retraining or alignment, improving its robustness while preserving the performance for downstream task-oriented decoding. Extensive experiments demonstrate that integrating FS consistently provides non-trivial certified robustness and significantly improves task-oriented performance under strong white-box adversarial attacks across diverse models and applications.
BibTeX
@misc{xia2026feature,
  author = {Song Xia and Meiwen Ding and Chenqi Kong and Wenhan Yang and Xudong Jiang},
  title = {{Feature-Space Smoothing: Certified Robustness of Deep Representations}},
  year = {2026},
  eprint = {2601.16200},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2601.16200},
}
Figure