Residue-Based Natural Language Adversarial Attack Detection

Vyas Raina, Mark J. F. Gales

Adversarial attacks—small, intentional perturbations designed to mislead machine learning models—have long been a primary concern in computer vision. However, as deep learning models increasingly dominate natural language processing (NLP), the nature of these attacks changes. Unlike the continuous pixel adjustments possible in images, text is discrete. This paper, presented at NAACL 2022, introduces Residue Detection, a method that turns the discrete nature of language from a vulnerability into a defensive advantage.

The Discrete Nature of Text Attacks

In the image domain, an attacker can modify a pixel's value by a tiny, nearly invisible amount. In contrast, NLP attacks typically involve swapping words, inserting characters, or deleting tokens. Because there is no "middle ground" between two distinct words in a vocabulary, these changes are inherently "loud."

The authors hypothesize that these discrete swaps leave a "residue" in the model's internal representations. While a model's encoder (e.g., BERT) is trained to compress semantic meaning into primary components, the disruptive "kick" of an adversarial swap leaks into other parts of the embedding space. This fundamental difference means that adversarial perturbations in text are more disruptive to internal representations than their image-based counterparts.

Methodology: Capturing the Residue

The proposed detection method focuses on the embedding of the [CLS] token. By applying Principal Component Analysis (PCA) to these embeddings, the researchers analyzed how signal is distributed across different eigenvalue ranks.

Figure 1: Comparison of average component size across eigenvalue ranks for original and attacked samples. The "residue" is visible as the gap between the two lines in the middle-tier components.

As shown in Figure 1, while the top components (representing core semantic features) remain relatively similar, a significant divergence appears in the "central" components. This "residue" is the signal the detector targets. The authors then train a simple linear classifier (logistic regression) on these embeddings to distinguish between clean and adversarial inputs. This approach is significantly more effective than "porting" image-based detectors like Mahalanobis Distance to the text domain.

Experimental Evaluation and Impact

The researchers tested Residue Detection across several benchmarks, including sentiment analysis (IMDB), emotion detection (Twitter), and topic classification (AG News and DBpedia). The detector demonstrated remarkable performance, achieving an F1-score of 0.91 on the IMDB dataset and 0.95 on AG News.

Crucially, the method proved specific to discrete data; when applied to the continuous image domain (CIFAR-100), the residue signal was absent, and detection performance plummeted. This confirmed the authors' hypothesis that the technique specifically exploits the "jumps" inherent in text manipulation.

A key measure of a detector's utility is its ability to reduce the "fooling rate"—the percentage of adversarial examples that successfully trick the model. Even when an adversary is aware of the detector and attempts to craft "stealthier" attacks, they are forced to make such minor changes that the attack's effectiveness is neutralized.

Figure 6: The impact of the residue detector on attack success. The "With" column shows a dramatic reduction in the fooling rate across all tested datasets.

As detailed in Figure 6, the presence of the residue detector slashed the fooling rate on the IMDB task from 0.70 to just 0.19. Similar drastic reductions were observed across all datasets, including regression tasks like L-Bus, where the impact score dropped from +0.51 to +0.23.

Conclusion

Residue Detection offers a computationally efficient and highly effective way to secure NLP models. By recognizing that text attacks cannot be infinitesimal, the authors provide a framework that identifies the "leftover" signals of adversarial manipulation. While currently a supervised method requiring pre-generated attack examples for training, it sets a new standard for NLP-specific defense strategies by exploiting the unique properties of discrete data.

Summary updated

Abstract

Deep learning based systems are susceptible to adversarial attacks, where a small, imperceptible change at the input alters the model prediction. However, to date the majority of the approaches to detect these attacks have been designed for image processing systems. Many popular image adversarial detection approaches are able to identify adversarial examples from embedding feature spaces, whilst in the NLP domain existing state of the art detection approaches solely focus on input text features, without consideration of model embedding spaces. This work examines what differences result when porting these image designed strategies to Natural Language Processing (NLP) tasks - these detectors are found to not port over well. This is expected as NLP systems have a very different form of input: discrete and sequential in nature, rather than the continuous and fixed size inputs for images. As an equivalent model-focused NLP detection approach, this work proposes a simple sentence-embedding "residue" based detector to identify adversarial examples. On many tasks, it out-performs ported image domain detectors and recent state of the art NLP specific detectors.
BibTeX
@inproceedings{raina2022residue,
  author = {Vyas Raina and Mark J. F. Gales},
  title = {{Residue-Based Natural Language Adversarial Attack Detection}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2022},
  doi = {10.18653/v1/2022.naacl-main.281},
}
Figure