In natural language, the relationship between two words often ignores their physical distance. In the sentence "The keys to the old, rusty cabinet are on the table," the plural verb are must agree with the subject keys, even though the singular noun cabinet sits right next to it.
This phenomenon, known as a long-distance dependency, is a fundamental test of whether a machine "understands" the hierarchical structure of language or is simply guessing based on the words it saw most recently. Research led by Yoav Goldberg and Tal Linzen has identified a specific stumbling block for neural networks: the syntactic attractor.
The Interference of Linear Proximity
An attractor is an intervening word—usually a noun—that carries different grammatical features (like number or gender) than the actual subject. For a model processing text linearly, these attractors act as "noise" that can overwrite the correct syntactic state.
As Tal Linzen and Yoav Goldberg demonstrated in Assessing the Ability of LSTMs to Learn Syntax-Sensitive Dependencies, Long Short-Term Memory (LSTM) networks are surprisingly good at subject-verb agreement when the sentence is simple. However, their performance degrades sharply when attractors are introduced. The model essentially "forgets" the subject's number because the attractor's features interfere with the information stored in the LSTM's hidden state.
This isn't just a problem for older recurrent models. Even modern Transformers like BERT, which use self-attention to look at all words simultaneously, face similar hurdles. While Yoav Goldberg noted in Assessing BERT's Syntactic Abilities that BERT handles these dependencies better than LSTMs, it still struggles when the hierarchical structure becomes deeply nested or when "nonsense" sentences strip away semantic clues.
Memory Decay vs. Structural Interference
Why do these errors happen? To understand this, researchers like Ethan Wilcox and Richard Futrell treat neural networks as psycholinguistic subjects. They found a key difference between how humans and AI fail:
- Humans struggle with long-distance dependencies primarily because of memory decay. The further apart the subject and verb are, the harder it is for us to keep the subject in mind.
- Artificial Models are less affected by distance itself but are highly sensitive to structural interference.
As Bingzhi Li and colleagues observed in How Distributed are Distributed Representations?, models often store syntactic information very locally. In French verb agreement, a model might discard the subject's number information almost immediately after the verb is processed. If an attractor appears in that short window, it can easily "hijack" the model's internal representation.
Building Structural Priors
If the problem is that models treat language as a flat sequence, the solution might be to force them to see the "tree" behind the words. Several researchers have proposed augmenting standard architectures with structural "common sense":
- Stack-Structured Memory: Dani Yogatama and Chris Dyer introduced a multipop adaptive stack for RNNs. By allowing the model to "push" and "pop" information, it can bypass intervening attractors, effectively jumping from the subject to the verb across the "noise" of a prepositional phrase.
- Dependency Path Embeddings: Instead of reading a sentence from left to right, Michael Roth and Mirella Lapata showed that embedding the path along a dependency tree allows models to capture relationships that are syntactically close but linearly distant.
- Hard-Coded Syntax: More recently, Yida Zhao and Kewei Tu developed Dependency Transformer Grammars, which use specialized attention masking to simulate a transition-based parser, ensuring the model's "focus" follows the actual grammatical structure of the sentence.
Why It Matters
The struggle with attractors isn't just a niche grammatical quirk; it's a proxy for how models handle complexity in the real world. In biomedical interaction extraction or news uncertainty identification, the "subject" and "action" are often buried under layers of clauses.
As Yair Lakretz and Stanislas Dehaene warn in Causal Transformers Perform Below Chance on Recursive Nested Constructions, even our most powerful models can be "brittle." Adding just a few words to a nested phrase can cause their performance to crash. Solving the attractor problem is therefore a prerequisite for moving beyond statistical surface-matching toward true linguistic competence.