Adversarial Attack for Explanation Robustness of Rationalization Models

Yuankai Zhang, Lingxiao Kong, Haozhao Wang, Ruixuan Li, Jun Wang, Yuhua Li, Wei Liu

Rationalization models are designed to enhance the interpretability of AI systems by providing "rationales"—subsets of input text that justify a model's prediction. For instance, in sentiment analysis, a model might highlight "the beer was flat" to explain a negative rating. While significant research has focused on improving the accuracy and plausibility of these rationales, their robustness against adversarial manipulation has been largely overlooked. This paper introduces UAT2E (Universal Adversarial Triggers to attack Explanations), a framework that exposes a critical vulnerability: models can be coerced into providing nonsensical or irrelevant explanations while maintaining their original, correct predictions.

Conceptual overview of adversarial attacks on explanation robustness. (a) Clean input yields a comprehensible explanation and correct prediction. (b) Crafted input yields an incomprehensible explanation while the prediction remains correct.

Methodology: The UAT2E Framework

The authors propose UAT2E, a gradient-based attack framework aimed at undermining explainability without altering the model's final output. Unlike traditional adversarial attacks that seek to change the model's label, UAT2E targets the "rationalizer" component. The attack operates in two primary modes:

  1. Non-target Attack: This mode aims to prevent the model from selecting the ground-truth or expected tokens, effectively "hiding" the real evidence.
  2. Target Attack: This mode forces the model to select specific adversarial triggers—often meaningless noise—as the rationale.

Technically, UAT2E utilizes a gradient-based search to identify vocabulary tokens that maximize the shift in the rationale while minimizing the change in prediction probability. To ensure efficiency in high-dimensional spaces, a KD-Tree is employed to find candidate tokens in the embedding space. The optimization objective is defined by a composite loss function that balances a Mean Squared Error (MSE) loss for rationale manipulation against a Cross-Entropy loss for prediction stability. This ensures that while the explanation is broken, the model's performance on the primary task remains ostensibly high.

Experimental Setup

The study evaluates five diverse datasets covering both token-level and sentence-level rationalization:

  • Movie: Binary sentiment classification.
  • FEVER: Fact verification (claim vs. document).
  • MultiRC: Multiple-choice question answering.
  • Beer & Hotel: Multi-aspect sentiment classification (e.g., rating a beer's "appearance" specifically).

These datasets were tested across various state-of-the-art rationalization architectures, including RNP, VIB, SPECTRA, FR, and DR.

Key Insights: Prediction vs. Explanation Robustness

A central finding of the research is the stark distinction between prediction robustness and explanation robustness. As illustrated in Figure 4, an attack on prediction aims to flip the label (e.g., from positive to negative), whereas an attack on explanation preserves the label but renders the justification meaningless.

Comparison of (a) original input, (b) adversarial attack on prediction, and (c) adversarial attack on explanation. Note how (c) maintains the 'positive' prediction but shifts the rationale to the adversarial trigger.

Other key insights include:

  • The BERT Paradox: Surprisingly, models using powerful encoders like BERT or those trained with human-annotated rationales (supervised learning) are actually more vulnerable to UAT2E. Their "cleaner" and more informative gradients allow the attack algorithm to more effectively identify the exact triggers needed to disrupt the explanation.
  • Degeneration and Spurious Correlations: Attacks often cause models to enter a state of "degeneration," where they select meaningless punctuation (e.g., commas, periods) as evidence. Alternatively, models may latch onto "spurious correlations," where unrelated words are treated as the primary justification for a score.
  • Transferability: Adversarial triggers discovered for one specific model frequently succeed against others, even in "black-box" scenarios where the attacker lacks knowledge of the target model's internal structure or gradients.

Results and Impact

Across all tested models and datasets, UAT2E successfully induced significant rationale shifts while keeping prediction accuracy within a 5% threshold of the original performance. In the Beer dataset, 57% of attacked samples suffered from degeneration. Furthermore, the "Attack Capture Rate" (AR)—the frequency with which the model selected the adversarial triggers themselves as the rationale—reached as high as 51.13% in some configurations. This means that in over half of the cases, the "explanation" provided to the user was nothing more than the attacker's injected noise.

Conclusion

This research highlights a significant gap in the development of explainable AI. While rationales are intended to build human trust by providing transparency, their fragility under adversarial conditions suggests that current XAI systems may provide a false sense of security. The paper demonstrates that a model can be "right for the wrong reasons" even when it appears to be explaining itself. Future work must move beyond simple extractive models and develop specific defense mechanisms, such as specialized adversarial training, to ensure that rationales remain grounded in genuine evidence, even under duress.

Summary updated

BibTeX
@inproceedings{zhang2024adversarial,
  author = {Yuankai Zhang and Lingxiao Kong and Haozhao Wang and Ruixuan Li and Jun Wang and Yuhua Li and Wei Liu},
  title = {{Adversarial Attack for Explanation Robustness of Rationalization Models}},
  booktitle = {ECAI},
  year = {2024},
  doi = {10.3233/faia240752},
}
Figure