TL;DR
This paper introduces XAI-Attack, a novel method for generating adversarial examples in text by leveraging explainable AI (XAI) techniques to identify and exploit model weaknesses. By pinpointing "falsely learned word indicators," XAI-Attack enables more effective adversarial training, leading to significantly more robust language models.
What Problem Does This Paper Address?
Machine learning models, especially in safety-critical domains like crisis informatics, can be easily fooled by adversarial examples – inputs subtly modified to cause incorrect predictions. While adversarial training aims to improve model robustness, current methods often yield only marginal gains due to ineffective evaluation and a lack of understanding of why models fail. Existing adversarial example generation techniques often require white-box access to the model or rely on simplistic perturbation strategies. This paper tackles the challenge of creating high-quality adversarial examples for black-box models that genuinely reveal and help rectify underlying model biases and "shortcut learning," where models rely on spurious correlations rather than true semantic understanding.
Methods
The core of XAI-Attack lies in its novel approach to generating adversarial examples. Instead of random perturbations or simple word substitutions, XAI-Attack uses XAI methods (specifically LIME and SHAP) as "importance functions."
- Identify Potential Adversarial Words: The method first identifies instances where the victim model makes incorrect predictions. For these misclassified instances, XAI techniques are applied to highlight words that are most indicative of the wrong prediction. These are termed "potential adversarial words."
- Filter Label-Invariant Words (Optional): To ensure the generated adversarial examples are semantically similar to the original, a filtering step is introduced. This step aims to remove words that, while flagged by XAI, might significantly alter the meaning of the sentence or are strong indicators of the correct class. Two filtering strategies are proposed: "count of label changes" and "indicator words for the correct class."
- Create Adversarial Examples: The filtered adversarial words are then inserted into correctly classified instances. The goal is to find insertions that flip the model's prediction without changing the original semantic meaning.
- Select Adversarial Examples: Instances whose classifications are successfully changed by the inserted words become the final adversarial examples.
This approach is coupled with adversarial training, where the model is retrained using these generated examples to improve its robustness. A key innovation is the use of XAI methods, which are flexible and can be easily swapped, and their application to black-box models using only hard labels.
Datasets
The experiments utilize several datasets from the GLUE benchmark: SST-2, RTE, QNLI, MNLI, and QQP. For adversarial testing and training, the paper also refers to the Adversarial GLUE benchmark, which consists of human-rated adversarial examples designed to be high-quality and valid.
Key Insights and Ideas
- XAI as a Weakness Detector: XAI isn't just for explaining correct predictions; it's a powerful tool for uncovering incorrect ones by highlighting spurious correlations learned by the model.
- Beyond Simple Perturbations: Generating adversarial examples by targeting specific "falsely learned word indicators" leads to more meaningful and effective attacks and training data.
- Shortcut Learning is Pervasive: Transformer models, despite their high performance on standard benchmarks, often rely on single words or simple patterns rather than complex reasoning, making them vulnerable.
- Robustness is Task-Dependent: Adversarial training with XAI-Attack shows significant improvements across various GLUE tasks, but the degree of improvement varies, highlighting the need for task-specific tuning.
- Transferability of Weaknesses: Adversarial examples generated for one model can often fool other transformer models, indicating shared vulnerabilities across architectures.
Key Results
- Improved Robustness: XAI-Attack, when used for adversarial training, significantly improves model robustness compared to baselines and other state-of-the-art methods. For instance, on the QQP dataset, XAI-Attack with no filtering achieved a 23.08 percentage point improvement in accuracy.
- High-Quality Adversarial Examples: Human evaluation confirmed that the generated adversarial examples are largely valid, meaning they change the model's prediction without altering the semantic meaning.
- Effective Adversarial Training: Models trained with XAI-Attack adversarial examples showed better performance on the Adversarial GLUE benchmark compared to models trained without or with other adversarial methods.
- Transferability: Adversarial examples generated by XAI-Attack on a distilBERT model were effective in deceiving other distilBERT, BERT, and RoBERTa models, demonstrating the generalizability of the identified biases.
Limitations and Future Work
- Model Size: The study primarily focused on smaller transformer models like distilBERT. Future work could explore XAI-Attack's effectiveness on larger models like GPT-3, which might be more robust but also potentially brittle with specific prompting.
- Hold-out Set Size: The method requires a hold-out set for generating adversarial examples. Further research is needed to determine the optimal size of this set and its impact on adversarial training.
- Insertion Strategy: While random insertion showed promise, more sophisticated methods for inserting adversarial words (e.g., using language models to predict optimal placement) could yield even better results and potentially imperceptible examples.
- Language Generalization: The current study is limited to English. Future work should investigate XAI-Attack's performance with other languages.
- Human Evaluation Nuances: The authors note that annotator performance in human evaluations was not perfect, potentially due to annotator fluency and the inherent difficulty of the tasks.
Summary updated