Abstract
Multimodal models, such as CLIP and BLIP, are increasingly integrated into critical AI systems but remain highly vulnerable to adversarial attacks that disrupt the alignment between disparate data modalities. This paper introduces a novel detection framework using Topological Data Analysis (TDA) to identify adversarial perturbations. By analyzing the "shape" of high-dimensional embeddings through persistent homology, we demonstrate that adversarial attacks leave distinct topological signatures characterized by "adversarial scattering." We propose two new metrics—Total Persistence (TP) loss and Multi-scale Kernel (MK) loss—and integrate them into statistical Maximum Mean Discrepancy (MMD) tests. Our results show that these topological features significantly improve the detection of stealthy adversarial examples, even in low-sample regimes where traditional statistical tests fail.
Introduction
The success of multimodal machine learning relies on the effective alignment of different data types, such as images and text, within a shared high-dimensional embedding space. However, this alignment creates a unique attack surface. Adversarial perturbations—often invisible to the human eye—can subtly shift an image's embedding, causing it to misalign with its corresponding text and leading to model failure.
Defending these systems is challenging because the misalignment occurs across different domains (e.g., pixels vs. tokens) that lack direct index alignment. Traditional unimodal defenses often fail to capture the global structural disruptions caused by these attacks. This research addresses the problem by treating the embedding space as a geometric point cloud and using TDA to detect the structural disorganization induced by adversaries.
Methodology: Topological Data Analysis
To capture the geometric nuances of the embedding space, the authors utilize persistent homology, a tool from TDA that tracks the birth and death of topological features (such as clusters, loops, and voids) across multiple scales.
Topological-Contrastive (TC) Losses
The framework introduces two primary metrics to quantify topological distortion:
- Total Persistence (TP) Loss: This provides a holistic summary of the cumulative importance of all topological features across all dimensions. It serves as a proxy for the overall "complexity" of the data distribution.
- Multi-scale Kernel (MK) Loss: This metric offers a robust comparison of topological features across various scales, allowing for a more granular analysis of how an adversary has shifted the data.
Enhanced Statistical Testing
These losses are integrated into the Maximum Mean Discrepancy (MMD) framework, a standard statistical method for determining if two datasets originate from the same distribution. By back-propagating the TC losses to the input samples, the authors generate gradients that act as "topological signatures." These signatures are then used in two new statistical tests: TPSAMMD and MKSAMMD. These tests are specifically designed to identify which samples in a batch contribute most to the topological scattering, making them highly sensitive to adversarial presence.
The Adversarial Scattering Phenomenon
A core insight of this work is the "Adversarial Scattering" assumption. The authors observed that as the concentration of adversarial examples in a batch increases, the Total Persistence (TP) loss increases monotonically.
In a clean dataset, image and text embeddings are tightly clustered due to their semantic alignment. Adversarial attacks disrupt this natural clustering, forcing embeddings toward incorrect labels and creating a more "scattered" and disorganized point cloud. The authors validated this intuition mathematically by modeling model logits as a Poisson Cluster Process. Monte Carlo simulations confirmed that increased scattering leads to higher topological complexity, specifically increasing the length of the Minimum Spanning Tree within the data.
Experimental Results
The proposed methods were evaluated against a rigorous matrix of 5 CLIP variants and 3 BLIP variants using datasets including CIFAR-10, CIFAR-100, and ImageNet. The tests were conducted against a variety of attacks, including PGD, AutoAttack (AA), and Carlini-Wagner (CW).
Detection Performance
- Superior Sensitivity: The TPSAMMD method consistently outperformed state-of-the-art two-sample tests. The advantage was most notable against stealthy, small-epsilon attacks (e.g., $\epsilon = 1/255$).
- Performance Gains: On ImageNet using CLIP ViT-L/14 under a PGD attack, TPSAMMD achieved an 8.6% gain in detection accuracy over the baseline SAMMD. For BLIP ViT-B/129, this gain reached 16.7%.
- Sample Efficiency: Unlike previous methods that require large batches (e.g., 500 samples) to detect statistical anomalies, the topological approach achieved near 100% test power with only 50 samples. This makes the defense viable for real-time applications where large batches are unavailable.
- Error Control: The tests maintained strict Type-I error control, keeping false positive rates at or below the 5% threshold.
Limitations and Future Directions
While effective, the computational cost of TDA is a known hurdle. Constructing Vietoris-Rips complexes for persistent homology is resource-intensive. To mitigate this, the authors utilized batch-processing approximations. Additionally, the researchers noted that very simple attacks like FGSM can sometimes produce non-monotonic topological signatures, likely because their massive, unoptimized distortions disrupt the point cloud in a fundamentally different way than sophisticated attacks.
Future work will focus on extending these topological signatures to other multimodal domains, such as video-text and audio-text systems, and optimizing the underlying TDA algorithms for broader deployment in AI safety pipelines.