Robustness and Adversarial Examples in Natural Language Processing

Kai-Wei Chang, He He, Robin Jia, Sameer Singh2021 · DOI

Abstract

Recent studies show that many NLP systems are sensitive and vulnerable to a small perturbation of inputs and do not generalize well across different datasets. This lack of robustness derails the use of NLP systems in real-world applications. This paper aims at bringing awareness of practical concerns about NLP robustness. It targets NLP researchers and practitioners who are interested in building reliable NLP systems. In particular, we will review recent studies on analyzing the weakness of NLP systems when facing adversarial inputs and data with a distribution shift. We provide a holistic view of: 1) how to use adversarial examples to examine the weakness of NLP models and facilitate debugging; 2) how to enhance the robustness of existing NLP models and defense against adversarial inputs; and 3) how the consideration of robustness affects the real-world NLP applications used in our daily lives.

The Illusion of Superhuman Performance

Modern Natural Language Processing (NLP) models frequently outperform humans on standard benchmarks and leaderboards. However, this "superhuman" performance is often a mirage. These models frequently rely on spurious correlations—statistical shortcuts that work on a specific dataset but fail in the real world. For example, a model might learn that the word "not" usually implies a negative sentiment in a specific training set, but it fails to understand the nuance of double negatives or sarcasm in a different context.

This lack of robustness is not merely an academic concern. In one real-world instance, a machine translation error—translating "Good morning" as "Attack them"—led to the wrongful arrest of a man by Israeli police. As NLP is integrated into high-stakes areas like content moderation, legal analysis, and translation, these "worst-case" failures become unacceptable. Average-case performance is a poor metric for safety-critical systems; we must instead test for egregious errors in seemingly benign messages.

Methodologies for Probing and Strengthening Models

To address these vulnerabilities, the field has developed three main pillars of research:

1. Adversarial Attack Generation

Techniques to "break" models are essential for debugging. These include:

  • Synonym Substitution: Replacing words with similar meanings to see if the model's prediction changes.
  • Syntactically Controlled Paraphrasing: Altering sentence structure while maintaining the original meaning.
  • Character-level Perturbations: Simulating typos or misspellings to test resilience against noisy input.

2. Defense and Mitigation

Once vulnerabilities are identified, researchers use several strategies to harden models:

  • Adversarial Training: Augmenting training data with adversarial examples so the model learns to ignore irrelevant perturbations.
  • Certified Robustness: Developing mathematical guarantees that a model’s prediction will remain constant within a certain range of input changes. This is particularly challenging in NLP because text is discrete; unlike computer vision, where pixels can be slightly adjusted, changing one word can fundamentally alter a sentence's logic.

3. Adversarial Data Collection

Static datasets are increasingly being replaced by "human-in-the-loop" processes. In benchmarks like Adversarial NLI (ANLI), humans actively try to find inputs that fool the model. These failures are then used to retrain and strengthen the system in an iterative cycle.

Key Insights and Landmark Findings

Research into robustness has yielded several critical insights into the nature of modern AI:

  • Accuracy ≠ Understanding: A model can achieve 99% accuracy on a test set while remaining "clueless" about underlying logic. Jia and Liang (2017) demonstrated this by adding a single irrelevant "distractor" sentence to a paragraph in the SQuAD dataset, which caused state-of-the-art model accuracy to plummet from 75% to 36%.
  • Adversarial Examples as Debuggers: Rather than seeing attacks as purely malicious, they should be viewed as essential tools for "behavior testing." If a model changes its classification because of a single typo, it has revealed a bug in its logical framework.
  • Universal Triggers: One of the more surprising findings is the existence of "triggers"—specific, often nonsensical sequences of tokens that, when added to any input, cause a model to produce a specific (often incorrect) prediction. This reveals deep-seated architectural vulnerabilities that are agnostic to the specific input text.

Future Directions

While the field is improving at "patching" specific vulnerabilities, a unified theory of NLP robustness remains elusive. Future work must address several key areas:

  • Scaling Certified Robustness: Current mathematical guarantees are computationally expensive and difficult to apply to massive Large Language Models (LLMs).
  • Natural vs. Adversarial Shifts: Systems must learn to distinguish between a malicious attacker and a "natural" shift in language, such as new slang, dialects, or evolving social contexts.
  • Accountability: There is a growing need for "accountable" NLP systems that can explain the reasoning behind a prediction, especially when that prediction is potentially erroneous.

Summary updated

Abstract

Recent studies show that many NLP systems are sensitive and vulnerable to a small perturbation of inputs and do not generalize well across different datasets. This lack of robustness derails the use of NLP systems in realworld applications. This tutorial aims at bringing awareness of practical concerns about NLP robustness. It targets NLP researchers and practitioners who are interested in building reliable NLP systems. In particular, we will review recent studies on analyzing the weakness of NLP systems when facing adversarial inputs and data with a distribution shift. We will provide the audience with a holistic view of 1) how to use adversarial examples to examine the weakness of NLP models and facilitate debugging; 2) how to enhance the robustness of existing NLP models and defense against adversarial inputs; and 3) how the consideration of robustness affects the real-world NLP applications used in our daily lives. We will conclude the tutorial by outlining future research directions in this area.
BibTeX
@misc{chang2021robustness,
  author = {Kai-Wei Chang and He He and Robin Jia and Sameer Singh},
  title = {{Robustness and Adversarial Examples in Natural Language Processing}},
  year = {2021},
  doi = {10.18653/v1/2021.emnlp-tutorials.5},
}
Figure