Abstract
Modern Natural Language Processing (NLP) models exhibit high confidence when classifying "rubbish examples"—nonsensical inputs that are unreadable to humans. This paper introduces Annealing Genetic-based Preposition Substitution (AGPS), an algorithm designed to expose model over-confidence by substituting meaningful words with prepositions. Unlike previous deletion-based methods, AGPS maintains the structural integrity of the input, leading to higher model confidence in nonsensical strings. Our results demonstrate that models like BERT and RoBERTa often rely on spurious statistical shortcuts, maintaining or even increasing confidence when presented with "word salads" of prepositions.
Introduction
Adversarial examples in NLP typically involve subtle perturbations that change a model's prediction. However, "rubbish examples" represent a different vulnerability: inputs so heavily modified they lose all human-intelligible meaning, yet the model remains "under-sensitive" to these changes.
Previous approaches, such as Input Reduction (IR), rely on iterative word deletion. IR suffers from three primary drawbacks: it frequently settles in local optima due to beam search limitations, it often leaves "semantic leaks" (single descriptive words that allow humans to infer the label), and it typically results in a significant drop in model confidence as the input length decreases.
Methodology: The AGPS Algorithm
To address these limitations, the authors propose the Annealing Genetic-based Preposition Substitution (AGPS) algorithm.
Preposition Substitution
Instead of deleting tokens, AGPS replaces nouns, verbs, adjectives, and adverbs with prepositions from a curated list of 37 (e.g., "of", "with", "at"). Prepositions lack independent semantic meaning but preserve the structural "bulk" of the sentence, preventing the model's internal representations from collapsing.
Annealing Genetic Algorithm (AGA)
The search for optimal substitutions is framed as an optimization problem solved via an Annealing Genetic Algorithm:
- Genetic Algorithm (GA): This mimics evolution by maintaining a population of modified sentences, using crossover and mutation to evolve effective rubbish examples.
- Simulated Annealing (SA): This incorporates the Metropolis principle, allowing the algorithm to occasionally accept inferior modifications. This stochasticity helps the search escape local optima, eventually finding global solutions with higher substitution rates and model confidence.
Experimental Results
The researchers evaluated AGPS across five benchmarks (SST-2, MR, SNLI, QQP, MRPC) against seven state-of-the-art models, including BERT and RoBERTa.
Key Findings
- Superior Obfuscation: AGPS achieved a 94.85% modification rate, successfully removing almost all semantic content while keeping predictions stable. As shown in the examples, while IR might leave a word like "joyful," AGPS replaces it with a preposition, creating a true "word salad."
- The Confidence Paradox: Models frequently assigned higher confidence to AGPS rubbish than to the original sentences. On average, AGPS produced examples with 97.17% confidence, compared to 94.67% for original text and 89.20% for IR.
- Shortcut Learning: The success of AGPS suggests that neural networks are not "reading" text in a human-like manner but are instead identifying persistent statistical patterns that remain even when semantic meaning is stripped away.
Conclusion and Future Work
AGPS exposes a critical vulnerability in modern NLP: the reliance on spurious shortcuts rather than deep linguistic understanding. The high transferability of these rubbish examples across different architectures (e.g., from BERT to ALBERT) suggests a shared underlying weakness in how these models process language.
While effective, the AGA approach is computationally intensive. Future research should focus on accelerating the generation process and applying these techniques to generative tasks. Furthermore, the authors demonstrated that "adversarial retraining"—incorporating these rubbish examples into the training set—can significantly improve model robustness against such vulnerabilities.
Summary updated