Expanding Scope: Adapting English Adversarial Attacks to Chinese

Hanyu Liu, Chengyuan Cai, Yanjun QiACL 2023

TL;DR

This paper tackles the challenge of extending English-based adversarial attack techniques to Chinese NLP models. By adapting existing methods and introducing Chinese-specific transformations (homophone and morphonym), the authors demonstrate that adversarial attacks can be effectively generalized to new languages, paving the way for more robust cross-lingual NLP systems.

What Problem Does This Paper Address?

Natural Language Processing (NLP) models, despite their impressive capabilities, are known to be vulnerable to adversarial attacks. These attacks involve making subtle, often imperceptible, changes to input text that can cause a model to misclassify or misinterpret the content. Historically, research in this area has heavily focused on English, utilizing language-specific resources like WordNet or English-pretrained BERT models. However, as NLP applications expand globally, there's a growing need to understand and improve the robustness of models for other languages, particularly those with different linguistic structures, like Chinese. This paper asks a crucial question: can state-of-the-art adversarial attack methods developed for English be successfully adapted to Chinese, and what new challenges and opportunities arise?

Methods

The core of this work is adapting existing English adversarial attack strategies to Chinese. The general approach involves:

  • Text Segmentation: Unlike English, Chinese doesn't use spaces to separate words. The authors employ a Chinese segmentation tool (Jieba) to break down sentences into meaningful word units, a crucial first step for word-level attacks.
  • Word Substitution: The primary attack strategy is word synonym substitution. This involves replacing words in a sentence with their synonyms to fool the target NLP model.
  • Search Strategy: To efficiently find effective substitutions, they use a greedy search algorithm that prioritizes words deemed "important" based on how much their removal impacts the model's confidence.
  • New Chinese-Specific Transformations: To better capture the nuances of the Chinese language, they introduce two novel attack types:
  • Homophone Transformation: Replaces characters with others that sound the same (share the same Pinyin pronunciation) but have different meanings.
  • Morphonym Transformation: Replaces characters with others that share similar visual components (pictographs), a common source of error for human writers.
  • Composite Attack: They combine multiple attack strategies (Open HowNet, Masked Language Model, Homophone, and Morphonym) into a single, more powerful attack.
  • Linguistic Constraints: To ensure the generated adversarial examples remain readable and semantically consistent, they employ constraints such as avoiding stop word modification, preventing repeated modifications, and using a Multilingual Universal Sentence Encoder (MUSE) to measure semantic similarity.

Datasets

The paper evaluates their methods on two Chinese datasets:

  • Online Shopping Review Dataset: Used for sentiment classification tasks with a BERT model.
  • Chinanews Dataset: A collection of news articles used for entailment tasks with a RoBERTa model.

These datasets are described as containing summaries and first paragraphs of news articles, labeled with various topic classes.

Key Insights and Ideas

  • Generalizability of English Attacks: The study demonstrates that English-based adversarial attack frameworks can indeed be adapted to Chinese, provided appropriate language-specific components (like segmentation and lexical resources) are integrated.
  • Importance of Language-Specific Nuances: The introduction of homophone and morphonym attacks highlights that exploiting language-specific features (phonology and morphology) can lead to highly effective adversarial examples that fool models even when semantic meaning is preserved for humans.
  • Homonym/Morphonym Attacks as a Double-Edged Sword: While these attacks are effective at fooling models, they can sometimes lead to less fluent or semantically consistent examples for human readers, suggesting a trade-off between attack success and naturalness.
  • Composite Attacks are Powerful: Combining different attack strategies significantly boosts the success rate, indicating that a multi-pronged approach is more robust against NLP models.
  • Adversarial Training is Effective: The paper shows that training models on these generated adversarial examples significantly improves their robustness against future attacks.

Key Results

  • High Attack Success Rates: The composite attack method achieved impressive success rates, reaching 96.00% for classification and 98.16% for entailment tasks.
  • Homonym and Open HowNet Shine: Among individual attacks, homonym-based and HowNet-derived methods showed better success rates than masked language model or morphonym-based attacks.
  • MUSE Constraint's Impact: The MUSE constraint, while improving quality, significantly decreased attack success rates for some methods, particularly Open HowNet and homonym substitutions.
  • Adversarial Training Boosts Robustness: Adversarial training reduced attack success rates by up to 49.32% for certain attack methods.
  • Human Evaluation: Homonym substitution yielded high consistency and fluency scores in human evaluations for classification tasks, while MLM was strong for entailment. Open HowNet generally received lower quality scores.

Limitations and Future Work

  • Language-Specific Efforts: The authors acknowledge that adapting the framework to languages significantly different from Chinese and English will require substantial effort in designing language-specific transformation methods.
  • Generalization Uncertainty: While optimistic about generalizing to other languages, they note uncertainty in achieving high attack success and fluency simultaneously for languages beyond Chinese and English.
  • Future Research: The paper suggests that the presented workflow can be generalized to build language-agnostic attack algorithms in future research. Further work could explore more sophisticated language-specific transformations and investigate the interplay between different linguistic features and adversarial robustness.

Summary updated

BibTeX
@inproceedings{liu2023expanding,
  author = {Hanyu Liu and Chengyuan Cai and Yanjun Qi},
  title = {{Expanding Scope: Adapting English Adversarial Attacks to Chinese}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
}
Figure