BERT & Family Eat Word Salad: Experiments with Text Understanding

Ashim Gupta, Giorgi Kvernadze, Vivek Srikumar

Introduction

State-of-the-art language models like BERT and RoBERTa have achieved superhuman performance on benchmarks like GLUE, leading to debates about whether they truly "understand" language or merely act as "stochastic parrots" exploiting spurious correlations. This paper investigates this by testing model responses to destructive transformations—perturbations that strip sentences of human-readable meaning, creating what the authors call "word salad."

Methodology

The researchers evaluated models using nine types of transformations across three categories:

  1. Lexical-overlap based: Shuffling, reversing, or alphabetically sorting tokens.
  2. Gradient-based: Identifying and dropping, repeating, or replacing "unimportant" tokens based on model gradients.
  3. Statistical (PBSMT): Using phrase-based statistical machine translation to generate sequences that mimic label-specific word patterns without grammatical structure.

Performance was measured by Agreement (how often the model's prediction on "word salad" matches the original) and Confidence (the probability assigned to the predicted label). The study utilized five GLUE datasets: MNLI, SNLI, MRPC, QQP, and SST-2.

Key Insights: The Word Salad Blind Spot

A primary finding is that models frequently force meaning onto invalid inputs. For instance, a model might classify an alphabetically sorted hypothesis as "entailment" with nearly the same confidence as the original coherent sentence, despite the input being nonsensical to humans.

Figure 1: An example of a "word salad" transformation where sorting the hypothesis alphabetically barely impacts the model's high-confidence entailment prediction.

This suggests that syntax is often optional for these models. In one experiment, a model trained exclusively on shuffled, nonsensical data performed nearly as well on coherent validation sets as a standard model. This implies that for many Natural Language Inference (NLI) tasks, models rely on a "bag of words" approach rather than syntactic structure.

Key Results

The models demonstrated alarming resilience to nonsense:

  • High Agreement: Across various datasets, models retained their original predictions on scrambled text in 70–90% of cases. For example, on MNLI, sorting the hypothesis alphabetically resulted in 79.1% agreement.
  • Extreme Confidence: The average confidence for predictions on invalid inputs exceeded 94%, significantly higher than random baselines (33% or 50%).
Figure 6: Agreement percentages across different datasets and transformations, showing how often models maintain their original predictions despite significant text degradation.

While human subjects could only guess labels for these inputs (roughly 35% accuracy), models maintained high accuracy and confidence. The most effective mitigation strategy found was training the model with an explicit "Invalid" class, allowing it to reject nonsense without degrading performance on legitimate language.

Conclusion

The study highlights a significant gap between model performance and true linguistic understanding. "Word salad" tests serve as a crucial diagnostic tool, revealing that high benchmark scores can mask a reliance on shallow statistical shortcuts. Future research should focus on training objectives that enforce respect for word order and syntactic structure to move beyond "stochastic parroting."

Summary updated

Abstract

In this paper, we study the response of large models from the BERT family to incoherent inputs that should confuse any model that claims to understand natural language. We define simple heuristics to construct such examples. Our experiments show that state-of-the-art models consistently fail to recognize them as ill-formed, and instead produce high confidence predictions on them. As a consequence of this phenomenon, models trained on sentences with randomly permuted word order perform close to state-of-the-art models. To alleviate these issues, we show that if models are explicitly trained to recognize invalid inputs, they can be robust to such attacks without a drop in performance.
BibTeX
@inproceedings{gupta2021bert,
  author = {Ashim Gupta and Giorgi Kvernadze and Vivek Srikumar},
  title = {{BERT \& Family Eat Word Salad: Experiments with Text Understanding}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2021},
  doi = {10.1609/aaai.v35i14.17531},
}
Figure