Introduction
State-of-the-art language models like BERT and RoBERTa have achieved superhuman performance on benchmarks like GLUE, leading to debates about whether they truly "understand" language or merely act as "stochastic parrots" exploiting spurious correlations. This paper investigates this by testing model responses to destructive transformations—perturbations that strip sentences of human-readable meaning, creating what the authors call "word salad."
Methodology
The researchers evaluated models using nine types of transformations across three categories:
- Lexical-overlap based: Shuffling, reversing, or alphabetically sorting tokens.
- Gradient-based: Identifying and dropping, repeating, or replacing "unimportant" tokens based on model gradients.
- Statistical (PBSMT): Using phrase-based statistical machine translation to generate sequences that mimic label-specific word patterns without grammatical structure.
Performance was measured by Agreement (how often the model's prediction on "word salad" matches the original) and Confidence (the probability assigned to the predicted label). The study utilized five GLUE datasets: MNLI, SNLI, MRPC, QQP, and SST-2.
Key Insights: The Word Salad Blind Spot
A primary finding is that models frequently force meaning onto invalid inputs. For instance, a model might classify an alphabetically sorted hypothesis as "entailment" with nearly the same confidence as the original coherent sentence, despite the input being nonsensical to humans.
This suggests that syntax is often optional for these models. In one experiment, a model trained exclusively on shuffled, nonsensical data performed nearly as well on coherent validation sets as a standard model. This implies that for many Natural Language Inference (NLI) tasks, models rely on a "bag of words" approach rather than syntactic structure.
Key Results
The models demonstrated alarming resilience to nonsense:
- High Agreement: Across various datasets, models retained their original predictions on scrambled text in 70–90% of cases. For example, on MNLI, sorting the hypothesis alphabetically resulted in 79.1% agreement.
- Extreme Confidence: The average confidence for predictions on invalid inputs exceeded 94%, significantly higher than random baselines (33% or 50%).
While human subjects could only guess labels for these inputs (roughly 35% accuracy), models maintained high accuracy and confidence. The most effective mitigation strategy found was training the model with an explicit "Invalid" class, allowing it to reject nonsense without degrading performance on legitimate language.
Conclusion
The study highlights a significant gap between model performance and true linguistic understanding. "Word salad" tests serve as a crucial diagnostic tool, revealing that high benchmark scores can mask a reliance on shallow statistical shortcuts. Future research should focus on training objectives that enforce respect for word order and syntactic structure to move beyond "stochastic parroting."
Summary updated