Introduction
A fundamental question in psycholinguistics is whether the human brain processes language as a linear sequence of words or through a hierarchical mental representation of sentence structure. This debate is often explored through the lens of "attraction errors"—grammatical slips where a nearby noun (the attractor) interferes with subject-verb agreement. For example, in the sentence "The girl near the boys likes climbing," the plural attractor "boys" might lure a speaker into using the plural verb "like."
This study investigates how the physical distance between an attractor and a verb influences agreement processing in both humans and state-of-the-art Neural Language Models (NLMs). By comparing human performance with models like GPT-3 and T5, the researchers sought to determine if AI architectures rely on human-like structural processing or purely linear statistical patterns.
Experimental Design and Methodology
The researchers utilized a "minimal triplet" design to isolate the effects of linear proximity versus structural hierarchy. Three primary conditions were established:
- Baseline: No attractor noun is present (e.g., "The writer who walks fast... climbs").
- Distal Attractor: The attractor is separated from the verb by intervening words (e.g., "The writer who knows the journalists the most climbs").
- Proximal Attractor: The attractor is immediately adjacent to the verb (e.g., "The writer who knows the happy young journalists climbs").
Participants and Models
Human participants (n=34) performed a violation detection task using Rapid Serial Visual Presentation (RSVP). In this setup, words are flashed sequentially to prevent re-reading, forcing the brain to process the sentence in real-time.
Two types of language models were tested:
- GPT-3 (via EleutherAI): Evaluated by comparing the probability assigned to correct vs. incorrect verb forms.
- T5: Fine-tuned specifically for grammatical error correction to provide a direct judgment comparable to humans.
Key Results: Humans vs. AI
The results revealed a stark contrast between human resilience and model vulnerability.
The Proximity Effect
For both humans and models, the "Proximal" condition proved the most challenging. As the attractor moved closer to the verb, error rates increased and human response times lengthened. This suggests that resolving the conflict between a linear distraction and structural reality requires significant cognitive effort.
Model Collapse and the "Bigram Lure"
The most striking finding was the performance of the T5 model in the proximal incongruent condition. When a plural attractor was placed directly before a singular verb, T5's accuracy dropped to near-chance levels (~50% error rate). This phenomenon, termed the "Bigram Lure," suggests that models over-rely on local transition probabilities (e.g., the common occurrence of a plural noun followed by a plural verb) rather than the global sentence structure.
The Human "Structural Safety Net"
In contrast, humans maintained high accuracy (rarely dropping below 80%) even in the most difficult proximal conditions. While humans are susceptible to linear interference, their processing appears primarily governed by a structural hierarchy that acts as a safety net against local distractions.
Discussion and Implications
The study highlights a significant gap between human linguistic competence and current AI capabilities. While Transformers like GPT-3 and T5 are highly sophisticated, their susceptibility to local attractors suggests they lack the robust structural representations inherent in human language processing.
Limitations and Future Directions
The authors identify several areas for future research:
- Semantic Content: The use of meaningful words might provide extra cues. Future studies using "Jabberwocky" sentences could isolate purely structural effects.
- Dependency Types: Expanding the research to gender agreement or reflexive pronouns would test the universality of these findings.
- Architecture Comparisons: Comparing Transformers to recurrent architectures like LSTMs might offer further insights into how sequential processing affects grammatical accuracy.
Ultimately, this research paves the way for developing more "structurally aware" AI that can better mimic the resilience of human language understanding.
Summary updated