TL;DR
This paper introduces "RedHerring," a novel adversarial attack designed to trick text attack detection models into flagging benign content as malicious, while leaving the original text classifier unaffected. This attack exploits a crucial vulnerability in current NLP security systems, aiming to erode human trust in automated detection mechanisms.
What Problem Does This Paper Address?
As AI-powered text classifiers become ubiquitous for content moderation on social media and other platforms, adversarial attacks have emerged to subvert them. While traditional adversarial attacks focus on making the classifier misclassify text (e.g., flagging hate speech as benign), this paper addresses a different threat: the unreliability of attack detection systems. These detectors are often used as a second line of defense, flagging potentially modified texts for human review. If these detectors frequently generate "false alarms" (flagging benign text as attacked), it can overwhelm human moderators, increase operational costs, and ultimately lead to a loss of trust in the entire detection system. The paper aims to explore and demonstrate this vulnerability.
Methods
The core of the paper is the RedHerring attack, a word-level adversarial attack with three key goals:
- Trigger the attack detector: The modified text must be classified as an "ATTACK" by the detector.
- Maintain classifier accuracy: The original text classifier must still assign the correct label to the modified text.
- Preserve human readability: The modified text should retain its original meaning and be understandable to a human reader.
The attack design involves:
- Selection: Identifying words in the original text that, when modified, are likely to influence the detector's output. This is done using a greedy approach that measures the change in the classifier's probability when a word is removed.
- Addition: Instead of pure replacement, RedHerring adds a mask token next to the selected word and uses a BERT-based model to suggest replacements. This aims to maintain semantic integrity while increasing the chance of triggering the detector. The attack iteratively checks suggestions, prioritizing those that flip the detector's label without changing the classifier's prediction.
Datasets
The experiments were conducted on four standard NLP datasets:
- AG News: Classification of news text into four categories (Sports, Business, World, Sci/Tech).
- IMDB: Binary sentiment classification on movie reviews.
- Rotten Tomatoes (RT): Binary sentiment classification on movie reviews.
- SST-2: Stanford Sentiment Treebank, containing binary sentiment classification on movie reviews.
The paper evaluates three attack detection algorithms (WDR, FGWS, UAPAD) against various classifiers (Albert, RoBERTa, DistilBERT, BERT, T5) on these datasets.
Key Insights and Ideas
- Shifting the Target: The most significant insight is the shift in adversarial attack focus from fooling the primary classifier to fooling the secondary attack detector. This highlights a new avenue for attackers to exploit NLP systems.
- Eroding Trust: RedHerring's success lies in creating a "false flag" scenario. When a human moderator sees a text flagged as an attack but correctly classified by the primary model, it directly undermines their confidence in the detector's reliability.
- Short Texts are More Vulnerable: The attack was found to be more effective against shorter texts, likely because modifications have a more pronounced effect on the overall text and are more easily detected by the greedy selection process.
- FGWS is Most Vulnerable: The FGWS detector was found to be the most susceptible to RedHerring, likely because it relies on replacing less frequent words, offering more opportunities for subtle manipulation.
- Confidence Matters: The paper demonstrates that the classifier's confidence score can be a valuable signal. RedHerring carefully avoids dropping this confidence, a key differentiator from attacks that aim to make the classifier fail.
Key Results
- Detection Accuracy Drops: RedHerring significantly reduced attack detection accuracy across all tested detectors and datasets. Drops ranged from 20 to 71 points. For example, FGWS saw an average drop of 62 points.
- Classifier Accuracy Maintained/Improved: Crucially, RedHerring maintained or even improved classifier accuracy. Average increases of up to 8 points were observed.
- Human Study Validation: Human annotators agreed with the detector's label only 41% of the time for RedHerring texts, compared to 67% for standard adversarial texts (PWWS) and 87% for clean texts, confirming the attack's ability to create distrust.
- BERTScore: Semantic integrity was largely preserved, with BERTScore values ranging from 80-98 for most datasets.
- Initial Defense Effectiveness: A simple "Confidence Check" defense, which leverages the classifier's confidence score, was proposed and tested. It significantly mitigated the attack, restoring up to 80% of detection accuracy without retraining.
Limitations and Future Work
- Limited Classifier Architectures: The study primarily used classifiers with similar architectures. Future work could explore RedHerring's effectiveness against more diverse model types.
- More Detection Algorithms and Datasets: The research tested three detection algorithms. Exploring a wider range of detectors and datasets could reveal more resilient systems or new vulnerabilities.
- LLM Robustness: While tested against T5, further investigation into RedHerring's impact on larger, more advanced language models is warranted.
- Arms Race: The paper acknowledges the ongoing "arms race" between attackers and defenders, suggesting that continued research into novel attacks and defenses is necessary. The authors also propose exploring randomness-based defenses and more sophisticated methods beyond the initial confidence check.
Summary updated