Deep Text Classification Can be Fooled

Bin Liang, Hongcheng Li, Miaoqiang Su, Pan Bian, Xirong Li, Wenchang ShiIJCAI 2018 · DOI

TL;DR

This paper demonstrates that deep neural networks (DNNs) used for text classification are surprisingly vulnerable to adversarial attacks, similar to their image counterparts. The authors introduce novel methods for crafting imperceptible yet effective adversarial text samples by identifying crucial text elements and strategically perturbing them, fooling state-of-the-art text classifiers.

What Problem Does This Paper Address?

While adversarial attacks on image and audio classification DNNs have been extensively studied, the vulnerability of DNNs in natural language processing (NLP) has been largely underestimated. Text classification is crucial for many applications like spam detection and phishing prevention. This paper addresses the gap by investigating whether text-based DNNs can be similarly fooled by adversarial samples and, if so, how to effectively craft such samples that are both deceptive to the model and imperceptible to humans, while preserving the original text's utility.

Methods

The authors propose a two-pronged approach for crafting adversarial text samples:

  • White-box Attack: When the target model's architecture and parameters are known, they leverage cost gradients from the input to identify "hot characters" and "hot phrases" (HTPs) that are most influential for classification. They then employ three perturbation strategies:
  • Insertion: Adding carefully selected HTPs, often disguised as facts or presuppositions, to alter the classification.
  • Modification: Slightly altering characters or words within HTPs (e.g., typos) to change the model's prediction.
  • Removal: Deleting non-essential words from influential phrases. They also introduce a "natural language watermarking" technique to embed these perturbations subtly.

  • Black-box Attack: When the model is opaque, they use a fuzzing-like technique. By systematically occluding words with whitespaces and observing how the classification changes, they can identify "hot sample phrases" (HSPs) that are critical for the current prediction. These HSPs are then used with the same perturbation strategies as in the white-box attack.

Datasets

The paper evaluates their methods on the following datasets:

  • DBpedia ontology dataset: A multi-class dataset with 560,000 training and 70,000 testing samples across 14 high-level classes.
  • MR (Movie Review) dataset: A movie review sentiment analysis dataset with 10,662 reviews.
  • CR (Customer Review) dataset: A dataset of 3,775 product reviews for sentiment analysis.
  • MPQA (Multi-Perspective Question Answering) dataset: A dataset of 10,606 opinions for subjectivity classification.

Key Insights and Ideas

  • Text DNNs are surprisingly vulnerable: The core finding is that, contrary to prior assumptions, text classification DNNs are as susceptible to adversarial attacks as image-based models.
  • Identifying "hot" text elements is key: The success of adversarial attacks hinges on precisely identifying which parts of the text are most critical for the classifier's decision.
  • Subtlety is paramount: Effective adversarial samples must not only fool the model but also remain imperceptible to humans and retain the original meaning and utility of the text.
  • Discrete nature of text poses unique challenges: Unlike continuous image data, text is discrete, making direct application of image-based perturbation methods ineffective and often leading to unnatural or meaningless text.
  • White-box vs. Black-box effectiveness: Both white-box and black-box attacks are shown to be effective, with significant overlap in identified influential phrases, suggesting complementary strengths.

Key Results

  • The proposed method successfully fools both character-level and word-level DNN-based text classifiers.
  • Adversarial samples can be perturbed to any desired class without compromising their utility.
  • In a user study, human observers could only detect modifications in 5.0% of cases, and the recall of where modifications occurred was only 2.0%, indicating the perturbations are difficult to perceive.
  • The white-box attack took approximately 116 hours for the DBpedia dataset, while the black-box attack took 107 hours, demonstrating reasonable efficiency for crafting adversarial samples.

Limitations and Future Work

  • Human effort required: The current method still requires some human effort in crafting adversarial samples, which might be a bottleneck for large-scale attacks or automated adversarial training.
  • Resource dependency for black-box: The black-box attack relies on access to training set data and confidence scores for feedback, which may not always be available.
  • Future directions: The authors suggest exploring automatic adversarial sample generation to support adversarial training and investigate large-scale attacks. They also mention the potential for adversarial samples to transfer between models, hinting at indirect black-box attack strategies. The paper also notes that while their method is effective, further research into defenses against such attacks is warranted.

Summary updated

BibTeX
@inproceedings{liang2018deep,
  author = {Bin Liang and Hongcheng Li and Miaoqiang Su and Pan Bian and Xirong Li and Wenchang Shi},
  title = {{Deep Text Classification Can be Fooled}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2018},
  doi = {10.24963/ijcai.2018/585},
}
Figure