Robustness-Aware Word Embedding Improves Certified Robustness to Adversarial Word Substitutions

Yibin Wang, Yichen Yang, Di He, Kun HeACL 2023 · OpenReview

Abstract

Deep learning models for natural language processing are highly susceptible to adversarial word substitutions, where replacing a few words with synonyms can flip a model's prediction. While Interval Bound Propagation (IBP) provides mathematical guarantees of robustness, its effectiveness is often limited by the "looseness" of bounds in standard embedding spaces. We present the Embedding Interval Bound Constraint (EIBC), a novel triplet loss-based approach that fine-tunes word embeddings to cluster synonyms tightly. This method significantly tightens the convex hulls used in IBP, leading to state-of-the-art certified robust accuracy on benchmarks like IMDB, YELP, and SST-2.

Introduction

The brittleness of NLP models under adversarial attacks poses a significant challenge for deployment in safety-critical applications. Most existing defenses offer only empirical robustness, which may fail against novel attack strategies. Certified robustness, conversely, provides a provable lower bound on model accuracy under any perturbation within a defined set.

Interval Bound Propagation (IBP) is a leading framework for certified robustness. It works by propagating a "bounding box" (convex hull) of all possible word substitutions through the network. However, in standard embeddings like GloVe, synonyms are often distant, resulting in large hulls that lead to overly conservative and loose bounds, ultimately degrading certified accuracy.

Methodology: Embedding Interval Bound Constraint (EIBC)

To address the embedding bottleneck, we propose the Embedding Interval Bound Constraint (EIBC). This "plug-and-play" solution fine-tunes the embedding layer using a specialized triplet loss before the main model training.

EIBC Triplet Loss shrinks the convex hull of word embeddings by pulling synonyms together and pushing non-synonyms apart.

The EIBC approach focuses on two primary objectives:

  1. Tightening the Hull: The loss function minimizes the maximum distance between a word and its synonyms across every dimension. This directly shrinks the interval of the convex hull that IBP must process.
  2. Preserving Semantic Structure: To avoid collapsing all words into a single point, a triplet structure is employed. While synonyms are clustered, non-synonyms are pushed apart until they reach a predefined safety margin ($\alpha$).

The training follows a two-stage process: first, embeddings are optimized via EIBC triplet loss; second, these robustness-aware embeddings are frozen while the rest of the model (e.g., a TextCNN) is trained using standard cross-entropy or IBP training.

Experimental Results

We evaluated EIBC on three sentiment analysis benchmarks: IMDB, YELP, and SST-2. Synonym sets were derived from counter-fitting GloVe clusters to ensure fair comparison with existing defenses.

Certified robust accuracy on IMDB and SST-2 datasets across different values of the parameter $\gamma$.

As shown in the figures above, the combination of EIBC and IBP training consistently outperforms standard IBP training across different parameter settings. On the IMDB dataset, certified robust accuracy improved from 76.73% to 84.78%. Similar gains were observed on SST-2 (+7.80%) and YELP (+4.94%).

Dataset Baseline (IBP) EIBC + IBP (Ours) Improvement
IMDB 76.73% 84.78% +8.05%
YELP 88.72% 93.66% +4.94%
SST-2 69.15% 76.95% +7.80%

Discussion and Key Insights

Beyond raw accuracy gains, EIBC offers several advantages:

  • Efficiency: Models using EIBC-pre-trained embeddings reach higher robustness levels in approximately half the training epochs required by standard IBP methods.
  • Generalization: The tightened embedding space naturally protects against unseen substitutions that fall within the same tight clusters.
  • Model Agnostic: Despite its simplicity, a TextCNN equipped with EIBC embeddings outperformed much larger BERT-based models using other certified defense techniques like SAFER or CISS.

Conclusion

EIBC demonstrates that the embedding layer is a critical factor in the certified robustness of NLP models. By explicitly constraining the interval of synonym clusters, we can provide significantly tighter bounds for IBP. Future work will focus on extending this approach to Transformer-based architectures and broader classes of adversarial perturbations beyond word substitutions.

Summary updated

BibTeX
@inproceedings{wang2023robustness,
  author = {Yibin Wang and Yichen Yang and Di He and Kun He},
  title = {{Robustness-Aware Word Embedding Improves Certified Robustness to Adversarial Word Substitutions}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
  url = {https://openreview.net/forum?id=8wz-INAVZG},
}
Figure