Enhancing Adversarial Defense via Brain Activity Integration Without Adversarial Examples

Takanori Nakajima, Keisuke Maeda, Ren Togo, Takahiro OgawaSensors 2025 · DOI

TL;DR

This paper introduces a novel adversarial defense method for vision-language models like CLIP that leverages human brain activity data. By integrating brain signals with augmented image features, the method achieves robust performance against adversarial attacks without requiring adversarial examples during training, thus overcoming the typical trade-off between robustness and clean image accuracy.

What Problem Does This Paper Address?

Large-scale vision-language models (LVLMs), such as those based on CLIP, are incredibly powerful but also vulnerable to adversarial attacks. These attacks subtly alter images, making them imperceptible to humans but causing models to misclassify them, severely degrading performance on downstream tasks. Traditional defense methods, like adversarial training, often require knowledge of the specific attack and lead to a performance drop on clean, unperturbed images. This creates a dilemma: a model can be robust to attacks or accurate on normal data, but not both. The authors aim to break this trade-off by developing a defense that is robust to unknown attacks and maintains high accuracy on clean data.

Methods

The core of the proposed method, termed "PM" (Proposed Method), lies in integrating human brain activity data with image processing. Instead of using adversarial examples, the approach utilizes functional magnetic resonance imaging (fMRI) data collected while subjects viewed original images.

Here's a breakdown:

  1. Augmented Image Set Creation: For each original image, a set of augmented images is generated using techniques inspired by SimCLR (e.g., cropping, resizing, flipping, rotation, grayscale, Gaussian blur). This aims to capture diverse visual features.
  2. Feature Extraction and Integration:
    • Visual features are extracted from the augmented images using a pre-trained CLIP image encoder.
  3. Brain features are extracted from the corresponding fMRI data using a specialized fMRI decoder (MindBridge).
  4. These visual and brain features are concatenated to form "integrated features."
  5. Training: A three-layer multilayer perceptron (MLP) is trained to predict the original visual features from the integrated features. The training objective is to maximize the cosine similarity between the predicted and original visual features. This encourages the model to learn features that are invariant to image perturbations, mimicking how humans perceive images despite minor changes.

Crucially, this method does not use adversarial examples during training, making it inherently robust to unseen attack types.

Datasets

  • Natural Scene Dataset (NSD) [36]: Used for both training and testing. This dataset contains pairs of original images (from MSCOCO) and corresponding fMRI data capturing human brain activity while viewing those images. The authors used 9841 samples from a single subject, with 8859 for training and 982 for testing.

Key Insights and Ideas

  • Brain Activity as a Robust Feature: The central hypothesis is that human brain activity exhibits invariant representations of objects and scenes, even when presented with slightly perturbed images. This inherent robustness of human perception can be harnessed to defend machine learning models.
  • Bypassing Adversarial Examples: The most significant insight is that effective adversarial defense can be achieved without explicitly training on adversarial examples. This circumvents the need to know the attack type and avoids the associated accuracy degradation on clean data.
  • Bridging Modalities: The method successfully integrates visual information from images with neural information from fMRI, creating a richer, more robust representation.
  • Human Perception as a Gold Standard: By aligning model outputs with human brain responses, the approach implicitly leverages the perceptual robustness of humans.

Key Results

  • Enhanced Robustness: The proposed method (PM) significantly improved robustness against various adversarial attacks (PGD, APGD, CW, Gao et al.) compared to baseline CLIP and other defense methods like Randomized Smoothing (RS) and Random Self-Ensemble (RSE). For instance, PM achieved average cosine similarities between 0.713 and 0.973 for adversarial examples, significantly higher than CLIP's (0.120-0.722).
  • Maintained Clean Accuracy: PM demonstrated excellent performance on clean images, achieving high cosine similarities (e.g., 0.973 for image similarity, 0.258 for text similarity), often outperforming even the baseline CLIP.
  • Attack Independence: PM showed consistent robustness across different attack types, even those it wasn't trained on, unlike "ideal" adversarial training methods (AT) which are attack-specific.
  • Effective Image-to-Text Retrieval: In image-to-text retrieval tasks, PM significantly outperformed baselines and even AT methods in retrieving correct captions for adversarial examples, while maintaining competitive performance on clean images. For example, PM achieved a Rank of 117.71 for PGD attacks, compared to CLIP's 494.31.

Limitations and Future Work

  • fMRI Data Impracticality: A major limitation acknowledged is the impracticality of using fMRI data during inference. fMRI is expensive, requires specialized equipment, and is difficult to obtain frequently, making real-time application challenging.
  • fMRI Decoder Specificity: The current fMRI decoder (MindBridge) is personalized, which limits its ease of use with new subjects and analysis across different brain regions.
  • Future Directions:
    • Multimodal Knowledge Distillation: Exploring training a single model with multiple modalities (fMRI, EEG, images) and transferring weights to enable inference using a single modality.
  • Improved fMRI Decoders: Developing simpler, more accurate fMRI decoders that are versatile and can analyze different brain regions.
  • Empirical Dataset Creation: Building a multisubject dataset for more comprehensive analysis of adversarial robustness with brain activity.

Summary updated

Abstract

Adversarial attacks on large-scale vision-language foundation models, such as the contrastive language-image pretraining (CLIP) model, can significantly degrade performance across various tasks by generating adversarial examples that are indistinguishable from the original images to human perception. Although adversarial training methods, which train models with adversarial examples, have been proposed to defend against such attacks, they typically require prior knowledge of the attack. These methods also lead to a trade-off between robustness to adversarial examples and accuracy for clean images. To address these challenges, we propose an adversarial defense method based on human brain activity data by hypothesizing that such adversarial examples are not misrecognized by humans. The proposed method employs an encoder that integrates the features of brain activity and augmented images from the original images. Then, by maximizing the similarity between features predicted by the encoder and the original visual features, we obtain features with the visual invariance of the human brain and the diversity of data augmentation. Consequently, we construct a model that is robust against adversarial attacks and maintains accuracy for clean images. Unlike existing methods, the proposed method is not trained on any specific adversarial attack information; thus, it is robust against unknown attacks. Extensive experiments demonstrate that the proposed method significantly enhances robustness to adversarial attacks on the CLIP model without degrading accuracy for clean images. The primary contribution of this study is that the performance trade-off can be overcome using brain activity data.
BibTeX
@inproceedings{nakajima2025enhancing,
  author = {Takanori Nakajima and Keisuke Maeda and Ren Togo and Takahiro Ogawa and Ren Togo},
  title = {{Enhancing Adversarial Defense via Brain Activity Integration Without Adversarial Examples}},
  booktitle = {Sensors},
  year = {2025},
  doi = {10.3390/s25092736},
}
Figure