BERT & Family Eat Word Salad: Experiments with Text Understanding

Ashim Gupta, Giorgi Kvernadze, Vivek Srikumar

Abstract

In this paper, we study the response of large models from the BERT family to incoherent inputs that should confuse any model that claims to understand natural language. We define simple heuristics to construct such examples. Our experiments show that state-of-the-art models consistently fail to recognize them as ill-formed, and instead produce high confidence predictions on them. As a consequence of this phenomenon, models trained on sentences with randomly permuted word order perform close to state-of-the-art models. To alleviate these issues, we show that if models are explicitly trained to recognize invalid inputs, they can be robust to such attacks without a drop in performance.

Introduction

State-of-the-art NLP models like BERT and RoBERTa have dominated benchmarks, often surpassing human performance. However, their true linguistic understanding remains a subject of debate. This research investigates a specific failure mode: insensitivity to destructive transformations. If a model truly understands a sentence, it should be confused when that sentence is turned into gibberish. Instead, we find that these models often "hallucinate" meaning, assigning high-confidence predictions to "word salad" inputs that lack syntax and semantics.

A model assigns 0.97 probability of entailment to a hypothesis with sorted word order, despite it being nonsensical.

Methodology

To test model robustness, we introduced three categories of Destructive Transformations designed to erase meaning while maintaining the task interface:

  1. Lexical Overlap-based: Operations such as sorting tokens alphabetically, reversing the sequence, or random shuffling. These preserve the "bag of words" but destroy grammatical structure.
  2. Gradient-based: Identifying "unimportant" tokens via model gradients and then dropping, repeating, or replacing them.
  3. Statistical (PBSMT): Using phrase-based statistical machine translation to generate sequences that mimic a specific label's co-occurrence statistics but remain ungrammatical.

We evaluated these transformations across five major GLUE datasets: MNLI, SNLI (Natural Language Inference), MRPC, QQP (Paraphrase Detection), and SST-2 (Sentiment Analysis).

Results and Analysis

Our experiments reveal a high degree of Agreement—the frequency with which a model's prediction on scrambled text matches its prediction on the original sentence. As shown in the table below, agreement scores frequently exceeded 80% across various tasks.

Agreement percentages between predictions on original and transformed text across GLUE benchmarks.

The "Bag of Words" Crutch

The high agreement on shuffled and sorted text suggests that models are largely ignoring word order, treating sentences as mere collections of tokens. In a provocative experiment, a model trained only on shuffled, meaningless data performed nearly as well on clean validation sets as models trained on standard English. This confirms that models are exploiting spurious statistical correlations rather than learning syntax.

Human vs. Machine Performance

There is a fundamental gap between human processing and transformer-based models. When humans were tasked with evaluating "word salad," their accuracy dropped to near-random levels (~35% for NLI). In contrast, RoBERTa models maintained an average confidence of over 90% on almost all destructive transformations, failing to recognize the inputs as invalid.

Mitigation and Conclusion

Standard calibration techniques like Label Smoothing failed to address this overconfidence. However, we found that adding an "Invalid" extra class during training was highly effective. By explicitly teaching the model to recognize nonsense, we achieved a 97%+ detection rate for word salad without harming performance on valid text.

While this study focused on encoder-based models like BERT and RoBERTa, the findings suggest a broader need for rigorous destructive testing in NLP. Future work should explore whether modern decoder-only LLMs exhibit similar vulnerabilities and seek generalized methods for models to "know what they don't know" without requiring explicit simulation of every possible nonsense type during training.

Summary updated

Abstract

In this paper, we study the response of large models from the BERT family to incoherent inputs that should confuse any model that claims to understand natural language. We define simple heuristics to construct such examples. Our experiments show that state-of-the-art models consistently fail to recognize them as ill-formed, and instead produce high confidence predictions on them. As a consequence of this phenomenon, models trained on sentences with randomly permuted word order perform close to state-of-the-art models. To alleviate these issues, we show that if models are explicitly trained to recognize invalid inputs, they can be robust to such attacks without a drop in performance.
BibTeX
@inproceedings{gupta2021bert,
  author = {Ashim Gupta and Giorgi Kvernadze and Vivek Srikumar},
  title = {{BERT \& Family Eat Word Salad: Experiments with Text Understanding}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2021},
  doi = {10.1609/aaai.v35i14.17531},
}
Figure