Pretrained Transformers Do not Always Improve Robustness

Swaroop Mishra, Bhavdeep Singh Sachdeva, Chitta BaralarXiv 2022 · OpenReview

Abstract

Pretrained Transformers (PTs) are widely celebrated for their superior Out-of-Distribution (OOD) robustness compared to traditional architectures. However, their performance in real-world settings involving noisy data remains a critical concern. This study evaluates 10 model architectures—ranging from Bag-of-Words (BOW) and LSTMs to state-of-the-art Transformers—and finds empirical evidence that PTs provide less robust representations than traditional models when exposed to poisoned training data. Furthermore, we demonstrate that adversarial filtering (AF) mechanisms, intended to improve generalization, are easily subverted by label noise.

Introduction

In real-world applications, datasets are rarely pristine. Data poisoning—where labels are incorrect due to annotator error, unclear instructions, or malicious intent—is an inevitable challenge. While recent research has lauded PTs like BERT and RoBERTa for handling distribution shifts, their susceptibility to noisy training data has been under-explored. We investigate whether the massive scale and self-supervised pre-training of Transformers help them ignore poisoned samples or, conversely, make them more prone to overfitting on noise.

Methodology

We conducted a comparative study across 10 architectures using sentiment classification benchmarks (SST-2 and IMDB). To simulate poisoning, we employed a label-flipping method, creating training sets with 0%, 30%, 50%, 70%, and 90% poisoned data.

To quantify sensitivity to noise, we introduced the Mean Rate of Change of Accuracy with Change in Poisoning (MRAP). A higher MRAP indicates a model that is more sensitive (less robust) to noise. We also developed AFPLite (Adversarial Filtering of Poisoned Data), an adaptation of AFLite designed to prune poisoned samples using RoBERTa embeddings before training.

Analysis of Robustness

Our experiments reveal a characteristic "U-shaped" accuracy curve across all models as poisoning levels increase. As noise approaches 50%, accuracy drops significantly as the model encounters maximum label conflict. However, beyond 50%, accuracy begins to increase again, indicating that the models effectively learn the "flipped" logic of the poisoned labels rather than ignoring them.

Accuracy across different levels of data poisoning (0% to 90%), showing the characteristic U-shaped curve as models learn flipped labels.

Crucially, we find that accuracy on clean data is a deceptive metric for robustness. Models that achieve the highest accuracy on clean benchmarks (Transformers) are often the most sensitive to noise. Simpler models like LSTMs and Bag-of-Words (BOW) exhibited higher stability across varying noise levels.

Key Results

The comparison between clean accuracy and the Normalized MRAP (NMRAP) highlights a significant trade-off. While Transformers dominate on clean datasets, they exhibit low robustness scores. Quantitatively, traditional models like LSTMs achieved a perfect 1.0 on the NMRAP scale (indicating maximum robustness), while Transformers scored as low as 0.14.

Comparison of normalized accuracy on clean data versus the Normalized Mean Rate of Change of Accuracy with Change in Poisoning (NMRAP) across various models.

This suggests that the quantity and diversity of pre-training data do not inherently protect a model from being "fooled" by noisy fine-tuning data. Furthermore, our AFPLite mechanism failed to reliably distinguish between poisoned and non-poisoned data, suggesting that the "shortcuts" Transformers find in data are present in both clean and noisy samples.

Conclusion

This study provides empirical evidence that Pretrained Transformers are less robust to noisy data than traditional models. Robustness to label noise is a distinct property from OOD generalization, and high performance on clean benchmarks does not guarantee resilience in the real world. Future research should focus on developing training objectives specifically designed to ignore poisoned samples rather than relying solely on the scale of pre-training.

Summary updated

Abstract

Pretrained Transformers (PT) have been shown to improve Out of Distribution (OOD) robustness than traditional models such as Bag of Words (BOW), LSTMs, Convolutional Neural Networks (CNN) powered by Word2Vec and Glove embeddings. How does the robustness comparison hold in a real world setting where some part of the dataset can be noisy? Do PT also provide more robust representation than traditional models on exposure to noisy data? We perform a comparative study on 10 models and find an empirical evidence that PT provide less robust representation than traditional models on exposure to noisy data. We investigate further and augment PT with an adversarial filtering (AF) mechanism that has been shown to improve OOD generalization. However, increase in generalization does not necessarily increase robustness, as we find that noisy data fools the AF method powered by PT.
BibTeX
@misc{mishra2022pretrained,
  author = {Swaroop Mishra and Bhavdeep Singh Sachdeva and Chitta Baral},
  title = {{Pretrained Transformers Do not Always Improve Robustness}},
  year = {2022},
  eprint = {2210.07663},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2210.07663},
}
Figure