Natural Adversarial Sentence Generation with Gradient-based Perturbation

Yu-Lun Hsieh, Minhao Cheng, Da-Cheng Juan, Wei Wei, Wen-Lian Hsu, Cho-Jui HsieharXiv 2019 · OpenReview

Abstract

This work proposes a novel algorithm to generate natural language adversarial input for text classification models, in order to investigate the robustness of these models. It involves applying gradient-based perturbation on the sentence embeddings that are used as the features for the classifier, and learning a decoder for generation. We employ this method to a sentiment analysis model and verify its effectiveness in inducing incorrect predictions by the model. We also conduct quantitative and qualitative analysis on these examples and demonstrate that our approach can generate more natural adversaries. In addition, it can be used to successfully perform black-box attacks, which involves attacking other existing models whose parameters are not known. On a public sentiment analysis API, the proposed method introduces a 20% relative decrease in average accuracy and 74% relative increase in absolute error.

Introduction

Adversarial attacks—inputs designed to induce model failure—are well-established in computer vision. In that domain, small perturbations to continuous pixel data can deceive a model without alerting a human observer. However, applying similar techniques to Natural Language Processing (NLP) is challenging because text is discrete. Existing NLP attacks often rely on "word-swapping," which frequently results in "word salad"—sentences that are ungrammatical or nonsensical to human readers. This research addresses the challenge of generating adversarial examples that are both effective at fooling models and "natural" enough to pass as human-written text.

Methodology

The proposed framework shifts the attack from the discrete word level to a continuous latent space. The architecture consists of two primary components: a classifier model and an attacker model.

The proposed framework for generating adversarial sentences.
  1. Architecture: An LSTM-based encoder-decoder model is employed. The encoder maps a sentence into a fixed-length vector (sentence representation), while the decoder is trained to reconstruct the original sentence from this vector.
  2. Gradient-based Perturbation: Instead of modifying words directly, the system calculates the gradient of the classifier's loss with respect to the sentence embedding. The Fast Gradient Sign Method (FGSM) is then used to perturb this embedding toward a target (incorrect) class.
  3. Natural Generation: The perturbed embedding is passed to the trained decoder. Since the decoder is trained on a large corpus of real text, it acts as a "naturalness filter," mapping the modified vector back into a grammatically coherent sentence.

Experimental Setup

The study utilizes the Yelp reviews polarity dataset, which contains 280,000 training samples and 19,000 test samples for binary sentiment classification (positive vs. negative). Byte-Pair Encoding (BPE) was used to manage a vocabulary of approximately 8,000 subwords.

Results and Discussion

The method demonstrates a significant improvement in the quality of adversarial text compared to baseline methods like the Jacobian-based saliency map attack (JSMA).

Quantitative Analysis

In human evaluations, the proposed method achieved a readability score of 0.63, significantly higher than the 0.37 achieved by JSMA. Fluency was also notably higher; the generated sentences had a log perplexity of 4.8, much closer to the original training data (4.2) than the JSMA method (6.1). Interestingly, while the sentences were more readable, human accuracy in identifying the intended sentiment dropped to 46.6%, suggesting the attack successfully "muddies the waters" for both machines and humans.

Black-Box Transferability

The attack's real-world viability was tested against the Amazon Comprehend API. Adversarial sentences generated via a surrogate model caused a 20% relative decrease in accuracy and a 74% increase in absolute error on the public API, proving that these examples transfer effectively to systems where internal parameters are unknown.

Qualitative Analysis

A key advantage of this approach is its ability to make holistic changes to a sentence's structure and tone. Unlike rule-based methods that simply swap single words, the gradient approach allows for a gradual polarity shift.

Qualitative comparison of adversarial sentences generated by the proposed method versus prior work.

As shown in the qualitative examples, the proposed model maintains grammatical structure and context far better than previous methods (e.g., Papernot et al. 2016), which often produce disjointed phrases or nonsensical word combinations.

Conclusion

This work demonstrates that perturbing latent sentence representations rather than discrete words allows for the generation of more natural and effective adversarial text. While challenges remain in making these attacks completely stealthy to suspicious readers, the integration of modern pre-trained language models offers a promising path for future improvements in fluency and task variety.

Summary updated

Abstract

This work proposes a novel algorithm to generate natural language adversarial input for text classification models, in order to investigate the robustness of these models. It involves applying gradient-based perturbation on the sentence embeddings that are used as the features for the classifier, and learning a decoder for generation. We employ this method to a sentiment analysis model and verify its effectiveness in inducing incorrect predictions by the model. We also conduct quantitative and qualitative analysis on these examples and demonstrate that our approach can generate more natural adversaries. In addition, it can be used to successfully perform black-box attacks, which involves attacking other existing models whose parameters are not known. On a public sentiment analysis API, the proposed method introduces a 20% relative decrease in average accuracy and 74% relative increase in absolute error.
BibTeX
@misc{hsieh2019natural,
  author = {Yu-Lun Hsieh and Minhao Cheng and Da-Cheng Juan and Wei Wei and Wen-Lian Hsu and Cho-Jui Hsieh},
  title = {{Natural Adversarial Sentence Generation with Gradient-based Perturbation}},
  year = {2019},
  eprint = {1909.04495},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1909.04495},
}
Figure