Introduction
In psycholinguistics, the "good-enough" theory posits that humans do not always perform an exhaustive syntactic analysis of every sentence. Instead, we often construct "shallow" representations that are sufficient for the immediate task, particularly under high cognitive or memory load. For instance, when encountering a sentence like "The ball kicked the boy," a listener might momentarily misinterpret the roles because "boys kicking balls" is a more plausible real-world scenario than the reverse.
While language models are often criticized for relying on surface-level heuristics rather than deep understanding, this research suggests that such behavior may actually be a hallmark of human-like processing. This paper investigates which architectural features—specifically the number of layers and attention heads—predispose a model to adopt these "good-enough" strategies.
The GELP Benchmark
To study this phenomenon, the authors introduced the Good-Enough Language Processing (GELP) benchmark. This dataset consists of 7,680 examples testing eight linguistic constructions, such as passive voice and dative shifts. Crucially, the benchmark varies "memory load" across three levels (one, two, or three propositions) to simulate the cognitive pressure that triggers heuristic processing in humans. The dataset was generated using templates and GPT-3.5 to ensure a wide variety of test cases.
Methodology
The study employed a two-pronged approach:
- Human Baseline: 304 English speakers were tested via crowdsourcing to establish how human accuracy fluctuates with memory load.
- Model Evaluation: 24 variants of BERT, ranging from 2 to 12 layers and 2 to 12 attention heads, were evaluated. These models were fine-tuned on standard Natural Language Inference (NLI) tasks (SNLI and MNLI) and the HANS dataset—the latter specifically designed to discourage simple lexical overlap shortcuts—before being tested on GELP.
The researchers utilized a "human-model matching score" to measure how closely a model's pattern of successes and failures aligned with human responses, rather than focusing solely on raw accuracy.
Key Findings
The Role of Model Depth
A primary insight from the study is that "good-enough" processing is inherently shallow. Increasing model depth beyond 8 layers yielded diminishing returns in matching human-like behavior. While the full BERT L12–A12 model achieved the highest human-model matching score at 74.3%, models with only 8 layers performed nearly as well. This suggests that the heuristics humans use are effectively captured by models with moderate depth.
Memory Load as a Trigger
Both humans and models exhibited a significant drop in accuracy as memory load increased. Humans averaged 86.6% accuracy overall, but this fell from 92.8% at low load to 80.9% at high load. The authors hypothesize that this fallibility stems from the encoding phase: when cognitive resources are taxed, the brain (and similarly, the model) fails to perform a detailed syntactic analysis to save energy, relying instead on semantic plausibility.
Attention Heads vs. Layers
Interestingly, while the number of layers had a clear impact on performance, the number of attention heads did not consistently correlate with human-like matching. This suggests that the retrieval mechanism associated with attention may not be the primary driver of these specific linguistic shortcuts.
Linguistic Blind Spots
Models struggled significantly more than humans with specific constructions. For example, in passive voice sentences, model accuracy plummeted to 43.5%. In these cases, models often ignored word order entirely, relying instead on simple word-meaning associations (e.g., associating "boy," "ball," and "kick" without respecting the syntactic structure of the passive construction).
Conclusion
The GELP benchmark demonstrates that human-like language processing is not always "perfect" processing. By showing that shallow models can mimic human heuristic-based errors, this research provides a new lens through which to evaluate Large Language Models. Future work will need to explore how models might be tuned to dynamically switch between robust algorithmic processing and efficient "good-enough" heuristics, much like the human mind balances accuracy and effort.
Summary updated