This paper, presented at the CLEF 2024 CheckThat! Lab, explores the fragility of misinformation detection systems. Researchers from the University of Zurich and Amsterdam UMC extended an adversarial attack algorithm called BeamAttack to see just how easily modern classifiers (like BERT and RoBERTa) can be tricked into misidentifying fake news or propaganda.
TL;DR
Researchers enhanced the BeamAttack algorithm with word deletion and LIME-based importance ranking, achieving a near-perfect (>99%) success rate in fooling misinformation classifiers. However, human evaluation revealed a major "semantic gap": while the models were easily tricked, many of the resulting adversarial texts became nonsensical or lost their original meaning to human readers.
What Problem Does This Paper Address?
Social media platforms rely heavily on machine learning to filter out harmful content, but these models have a "glass jaw." Adversarial attacks—small, intentional perturbations to input text—can cause a model to flip its prediction (e.g., labeling "fake news" as "credible").
Most existing attacks use "greedy search," changing words one by one until the model breaks. This often leads to sub-optimal, bulky changes. The authors address the need for more sophisticated, "grey-box" evaluation tools that can find the minimal possible changes required to deceive a model while attempting to keep the text readable for humans.
Methods
The authors build upon BeamAttack, a search-based attack that explores multiple potential modifications simultaneously (a "beam") rather than just the single best immediate change.
Key innovations introduced in this paper include:
- Word Deletion: Unlike previous versions that only substituted words, this version can choose to remove words entirely if it helps trigger a misclassification.
- LIME Integration: Instead of just looking at raw model outputs (logits), they integrated LIME (Local Interpretable Model-agnostic Explanations) to identify which specific tokens most heavily influence a model's decision for a particular sentence.
- Contextual Substitutions: They used a Masked Language Model (RoBERTa) to suggest replacement words that fit the grammatical context of the sentence.
- Beam Search Optimization: They tuned hyperparameters like beam size ($k$) and branching factor ($b$) to balance attack success against the number of queries made to the victim model.
Datasets
The study utilized the BODEGA framework, which includes five distinct misinformation detection tasks:
- HN (News Bias): Categorizing articles based on stylistic cues.
- PR2 (Propaganda): Identifying snippets using persuasive techniques.
- FC (Fact Checking): Verifying news against Wikipedia snippets.
- RD (Rumor Detection): Analyzing informal Twitter threads.
- C19 (COVID-19): Detecting misinformation specifically about the pandemic.
Key Insights and Ideas
- The Robustness Hierarchy: As expected, simpler models like BiLSTM were the easiest to break. BERT was more resilient, and adversarially trained RoBERTa was the toughest, requiring much larger "beams" and more queries to find a successful attack.
- LIME vs. Logits: For complex tasks like Fact Checking (FC), using LIME to find "important" words was more effective than standard logit-based methods, though it required significantly more computational power.
- The Semantic Trade-off: There is a clear tension between "fooling the machine" and "fooling the human." The attack was highly successful at the former but struggled with the latter.
- Syntactic Stability: Interestingly, in nearly 44% of successful single-word attacks, the Part-of-Speech (POS) tag of the word didn't change. The attack often swaps a noun for a noun or a verb for a verb, maintaining the "shape" of the sentence even if the meaning drifts.
Key Results
- Success Rate: The extended BeamAttack achieved an attack success rate of over 99% across multiple datasets and victim models.
- Performance: It outperformed established baselines like BERT-ATTACK and DeepWordBug on BERT and BiLSTM victims.
- Human Evaluation Failure: This is the most striking result. Despite high automated scores (like BLEURT), human evaluators found that only 7% of adversarial samples actually preserved the original meaning. 30% were deemed nonsensical, largely due to the new word-deletion feature breaking sentence structure.
- Query Intensity: The attack is "expensive." For some tasks, it required double the number of model queries compared to genetic algorithm-based attacks.
Limitations and Future Work
The authors identify several "failure modes" for BeamAttack:
- Grammar Breakdowns: Removing words often leads to syntactically "broken" sentences that a human would immediately flag as suspicious.
- Semantic Inversion: The Masked Language Model often suggests antonyms. While this effectively tricks the classifier, it changes the meaning of the text (e.g., turning a "lie" into a "truth"), which defeats the purpose of a "stealthy" adversarial attack.
- Computational Cost: The reliance on LIME and large beam widths makes the attack slow and difficult to scale without significant GPU resources.
Future Work: The authors suggest using Large Language Models (LLMs) to generate more fluent substitutions and implementing stricter "similarity filters" during the search process to ensure the adversarial text remains meaningful to humans.
Summary updated