Michael Lepori, Tal Linzen, R. Thomas McCoy. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. 2020.
Introduction
While sequential neural networks like LSTMs are surprisingly effective at capturing language patterns, they often struggle with the hierarchical nature of syntax. Human language is not merely a string of words but a nested hierarchy. For instance, in the sentence "The boys by the red truck kick the ball," a simple sequential model might associate the singular "truck" with the verb, leading to the error "truck kicks." However, the true subject is the plural "boys."
This paper investigates whether "hard-wiring" models to process sentences according to their tree structure—specifically constituency and dependency structures—makes them more robust against such superficial statistical shortcuts.
Architectures and Methods
The researchers compared four distinct architectures to evaluate the impact of structural priors:
- BiLSTM (Baseline): A standard sequential model processing sentences forward and backward.
- Constituency LSTM: A tree-based model following a binary constituency parse.
- Dependency LSTM: A tree-based model processing words based on head-child relationships.
- Head-Lexicalized Tree LSTM: A hybrid using constituency structure but lexicalizing internal nodes with the "head" word of the phrase.
The models were tasked with number prediction: identifying whether a masked main verb in a sentence (e.g., "The girl MASK the ball") should be singular or plural.
Evaluation Framework
The models were tested across several datasets:
- Natural Language (Wikipedia): A balanced subset of the Linzen et al. (2016) dataset.
- Adversarial Test Sets: Sentences containing "attractors"—nouns with the opposite number placed between the subject and verb.
- Constructed (PCFG) Dataset: A synthetic dataset where words were equally likely in any position, preventing models from relying on "common sense" or semantic co-occurrence.
- Augmentation Set: A small 500-sentence synthetic set used for fine-tuning to see if sequential models could learn structural biases.
Key Results: The Attractor Gap and Heuristic Resistance
The study revealed significant performance disparities when models faced complex syntax. On natural language sentences with four "attractor" nouns, the standard BiLSTM's accuracy plummeted to 41%, performing worse than chance. In contrast, the Dependency and Constituency models maintained much higher accuracies of 87% and 80.8%, respectively.
A critical finding was the "Dependency Trap." While Dependency LSTMs performed well on natural language, their accuracy dropped to 73.8% on the synthetic "constructed" set. This suggests that without explicit labels, dependency models rely heavily on word co-occurrence heuristics—guessing the subject based on semantic plausibility rather than pure syntax. Constituency models, however, remained nearly perfect at 97.6%, proving to be the most robust against statistical shortcuts.
The Power of Fine-Tuning
One of the most striking results was the impact of minimal structural data on the sequential baseline. After fine-tuning on just 500 syntactically rich sentences from the augmentation set, the BiLSTM's performance on the 4-attractor natural language set jumped from 41% to 81.1%.
This suggests that sequential models possess the capacity to learn these structural biases; they simply do not receive a sufficiently strong signal from standard natural language corpora.
Conclusion and Future Work
The research highlights that constituency-based models are the "gold standard" for structural robustness. However, these models require ground-truth syntactic parses, which are not always available. The authors suggest that future work should explore unsupervised tree induction, enabling models to discover hierarchical structures autonomously. Additionally, adding "typed" dependencies (e.g., distinguishing subjects from objects) might help dependency-based models bridge the gap with constituency-based architectures.
Summary updated