Abstract
Recurrent neural networks (RNNs) have achieved impressive results in a variety of linguistic processing tasks, suggesting that they can induce non-trivial properties of language. We investigate here to what extent RNNs learn to track abstract hierarchical syntactic structure. We test whether RNNs trained with a generic language modeling objective in four languages (Italian, English, Hebrew, Russian) can predict long-distance number agreement in various constructions. We include in our evaluation nonsensical sentences where RNNs cannot rely on semantic or lexical cues ("The colorless green ideas sleep furiously"), and, for Italian, we compare model performance to human intuitions. Our language-model-trained RNNs make reliable predictions about long-distance agreement, and do not lag much behind human performance. We thus bring support to the hypothesis that RNNs are not just shallow-pattern extractors, but they also acquire deeper grammatical competence.
Introduction
A central debate in natural language processing is whether Recurrent Neural Networks (RNNs), such as LSTMs, truly grasp the hierarchical syntactic structure of language or if they simply exploit shallow linear patterns and semantic co-occurrences. While LSTMs excel at language modeling, previous research suggested they might struggle with complex syntax—like long-distance subject-verb agreement—unless explicitly supervised for those tasks. Furthermore, it has been difficult to determine if models use semantic cues (e.g., "dogs" and "bark") rather than structural rules to make predictions. This study introduces a rigorous framework to isolate pure syntactic competence by removing semantic and frequency-based confounds.
Methodology
To evaluate syntactic understanding independently of semantics, the authors utilized "nonce" sentences—grammatically correct but semantically nonsensical strings, inspired by Noam Chomsky’s famous example, "Colorless green ideas sleep furiously."
The researchers automatically extracted long-distance number agreement constructions from dependency treebanks in Italian, English, Hebrew, and Russian. In these constructions, a "cue" (e.g., a subject) and a "target" (e.g., a verb) are separated by at least three words. To create nonce variants, content words were replaced with random words sharing the same part-of-speech and morphological features.
Standard LSTMs were trained on 90-million-token Wikipedia subsets for each language and evaluated on their ability to assign higher probability to the grammatically correct form of a target word. Performance was compared against several baselines, including n-gram models and human judgments collected for the Italian dataset.
Key Results and Insights
The study reveals that LSTMs are capable of inducing abstract grammatical rules solely from raw text, without relying on semantic memorization.
Performance on Nonce Sentences
The LSTMs significantly outperformed all baselines on nonce sentences, where semantic context is absent. Accuracy reached 85.5% in Italian, 74.1% in English, 80.8% in Hebrew, and 88.8% in Russian. The success on these semantically void sentences proves that the models are not merely memorizing word co-occurrences but are tracking structural dependencies.
Comparison with Human Performance
In the Italian evaluation, the LSTM's performance on nonce sentences (85.5%) was remarkably close to that of native speakers (88.4%). Notably, the models and humans tended to struggle with the same complex linguistic configurations, suggesting a shared sensitivity to syntactic complexity.
Robustness to Attractors
"Attractors"—intervening words with opposite grammatical number (e.g., "The girl with the dogs...")—often mislead simpler models. However, well-tuned LSTMs demonstrated significant robustness to these distractors, with performance degradation patterns similar to those observed in humans as the number of attractors increased.
The Role of Morphology and Perplexity
Models performed better in morphologically rich languages like Italian and Russian compared to English, which suffers from higher part-of-speech ambiguity. Additionally, a strong correlation was found between a model's overall validation perplexity and its syntactic accuracy, suggesting that syntactic competence is a natural byproduct of optimizing for general word prediction.
Conclusion
This research provides strong evidence that RNNs can acquire deep grammatical competence without explicit syntactic supervision or semantic cues. While English and Hebrew presented specific challenges likely due to structural ambiguities, the overall results suggest that hierarchical processing is an emergent property of large-scale language modeling. Future work will focus on analyzing the internal hidden states of these networks to further decode how they represent syntactic information.
Summary updated