Abstract
This work proposes a novel algorithm to generate natural language adversarial input for text classification models, in order to investigate the robustness of these models. It involves applying gradient-based perturbation on the sentence embeddings that are used as the features for the classifier, and learning a decoder for generation. We employ this method to a sentiment analysis model and verify its effectiveness in inducing incorrect predictions by the model. We also conduct quantitative and qualitative analysis on these examples and demonstrate that our approach can generate more natural adversaries. In addition, it can be used to successfully perform black-box attacks, which involves attacking other existing models whose parameters are not known. On a public sentiment analysis API, the proposed method introduces a 20% relative decrease in average accuracy and 74% relative increase in absolute error.
Introduction
Adversarial attacks—inputs designed to induce model failure—are well-established in computer vision. In that domain, small perturbations to continuous pixel data can deceive a model without alerting a human observer. However, applying similar techniques to Natural Language Processing (NLP) is challenging because text is discrete. Existing NLP attacks often rely on "word-swapping," which frequently results in "word salad"—sentences that are ungrammatical or nonsensical to human readers. This research addresses the challenge of generating adversarial examples that are both effective at fooling models and "natural" enough to pass as human-written text.
Methodology
The proposed framework shifts the attack from the discrete word level to a continuous latent space. The architecture consists of two primary components: a classifier model and an attacker model.
- Architecture: An LSTM-based encoder-decoder model is employed. The encoder maps a sentence into a fixed-length vector (sentence representation), while the decoder is trained to reconstruct the original sentence from this vector.
- Gradient-based Perturbation: Instead of modifying words directly, the system calculates the gradient of the classifier's loss with respect to the sentence embedding. The Fast Gradient Sign Method (FGSM) is then used to perturb this embedding toward a target (incorrect) class.
- Natural Generation: The perturbed embedding is passed to the trained decoder. Since the decoder is trained on a large corpus of real text, it acts as a "naturalness filter," mapping the modified vector back into a grammatically coherent sentence.
Experimental Setup
The study utilizes the Yelp reviews polarity dataset, which contains 280,000 training samples and 19,000 test samples for binary sentiment classification (positive vs. negative). Byte-Pair Encoding (BPE) was used to manage a vocabulary of approximately 8,000 subwords.
Results and Discussion
The method demonstrates a significant improvement in the quality of adversarial text compared to baseline methods like the Jacobian-based saliency map attack (JSMA).
Quantitative Analysis
In human evaluations, the proposed method achieved a readability score of 0.63, significantly higher than the 0.37 achieved by JSMA. Fluency was also notably higher; the generated sentences had a log perplexity of 4.8, much closer to the original training data (4.2) than the JSMA method (6.1). Interestingly, while the sentences were more readable, human accuracy in identifying the intended sentiment dropped to 46.6%, suggesting the attack successfully "muddies the waters" for both machines and humans.
Black-Box Transferability
The attack's real-world viability was tested against the Amazon Comprehend API. Adversarial sentences generated via a surrogate model caused a 20% relative decrease in accuracy and a 74% increase in absolute error on the public API, proving that these examples transfer effectively to systems where internal parameters are unknown.
Qualitative Analysis
A key advantage of this approach is its ability to make holistic changes to a sentence's structure and tone. Unlike rule-based methods that simply swap single words, the gradient approach allows for a gradual polarity shift.
As shown in the qualitative examples, the proposed model maintains grammatical structure and context far better than previous methods (e.g., Papernot et al. 2016), which often produce disjointed phrases or nonsensical word combinations.
Conclusion
This work demonstrates that perturbing latent sentence representations rather than discrete words allows for the generation of more natural and effective adversarial text. While challenges remain in making these attacks completely stealthy to suspicious readers, the integration of modern pre-trained language models offers a promising path for future improvements in fluency and task variety.
Summary updated