Representations of Syntax [MASK] Useful: Effects of Constituency and Dependency Structure in Recursive LSTMs

Michael A. Lepori, Tal Linzen, R. Thomas McCoy

Abstract

Sequence-based neural networks show significant sensitivity to syntactic structure, but they still perform less well on syntactic tasks than tree-based networks. Such tree-based networks can be provided with a constituency parse, a dependency parse, or both. We evaluate which of these two representational schemes more effectively introduces biases for syntactic structure that increase performance on the subject-verb agreement prediction task. We find that a constituency-based network generalizes more robustly than a dependency-based one, and that combining the two types of structure does not yield further improvement. Finally, we show that the syntactic robustness of sequential models can be substantially improved by fine-tuning on a small amount of constructed data, suggesting that data augmentation is a viable alternative to explicit constituency structure for imparting the syntactic biases that sequential models are lacking.

Introduction

Natural language is inherently hierarchical, not sequential. A classic example is English subject-verb agreement: a verb must match the number of its subject, regardless of how many other nouns appear between them. Sequential neural networks, like standard LSTMs, process text left-to-right and often fall back on shallow heuristics—such as matching the verb to the most recently seen noun. This paper investigates whether neural networks need explicit syntactic structures (like tree parses) to robustly learn these rules, or if sequential models can learn them from raw text with the right data.

Methodology

The authors frame subject-verb agreement as a masked word prediction task. They compare four architectures:

  1. BiLSTM: A standard sequential baseline.
  2. Constituency LSTM: A Tree-LSTM using binary constituency parse trees.
  3. Dependency LSTM: A Tree-LSTM that sums representations of a headword's children.
  4. Head-Lexicalized Tree LSTM: A novel hybrid model using a constituency backbone with dependency headword information.

Models were trained on a balanced Wikipedia subset and evaluated on two test sets: a natural language set with varying numbers of adversarial attractors, and a constructed set (PCFG-generated) designed to remove natural word co-occurrence clues, forcing a reliance on pure syntax.

Results: Structural Biases

Initial evaluations revealed that tree-based models are significantly more robust to adversarial attractors in natural language than the sequential BiLSTM. However, a critical failure was observed for the Dependency LSTM on the constructed PCFG dataset. Because it sums child nodes and discards linear order, it relies heavily on lexical heuristics that fail when those patterns are absent. In contrast, the Constituency LSTM maintained high accuracy, demonstrating a more robust syntactic bias.

Figure 2: Performance on natural and constructed evaluation sets before data augmentation. The sequential BiLSTM struggles with attractors, while the Dependency LSTM fails on the constructed set.

Results: The Power of Data Augmentation

A key finding of this research is that architectural bias may not be the only path to syntactic robustness. By fine-tuning the baseline BiLSTM on just 500 syntactically challenging, artificially generated sentences, its performance on the natural test set with four attractors jumped from 41.0% to 81.1%. This suggests that sequential models have the capacity to learn hierarchical rules when provided with data that explicitly breaks shallow heuristics.

Figure 4: Performance on natural and constructed evaluation sets after fine-tuning on augmented data. Note the significant improvement in the BiLSTM's robustness to attractors.

Conclusion

While explicit constituency structure provides the most robust generalization out-of-the-box, data augmentation offers a powerful and flexible alternative for imparting syntactic biases to sequential models. Combining constituency and dependency structures did not yield further gains, suggesting that the primary benefit comes from the hierarchical grouping provided by constituency trees. Future work should investigate whether these findings extend to a broader range of linguistic phenomena beyond subject-verb agreement.

Summary updated

Abstract

Sequence-based neural networks show significant sensitivity to syntactic structure, but they still perform less well on syntactic tasks than tree-based networks. Such tree-based networks can be provided with a constituency parse, a dependency parse, or both. We evaluate which of these two representational schemes more effectively introduces biases for syntactic structure that increase performance on the subject-verb agreement prediction task. We find that a constituency-based network generalizes more robustly than a dependency-based one, and that combining the two types of structure does not yield further improvement. Finally, we show that the syntactic robustness of sequential models can be substantially improved by fine-tuning on a small amount of constructed data, suggesting that data augmentation is a viable alternative to explicit constituency structure for imparting the syntactic biases that sequential models are lacking.
BibTeX
@inproceedings{lepori2020representations,
  author = {Michael A. Lepori and Tal Linzen and R. Thomas McCoy},
  title = {{Representations of Syntax [MASK] Useful: Effects of Constituency and Dependency Structure in Recursive LSTMs}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2020},
  url = {https://openreview.net/forum?id=75NqS0JJr5c},
}
Figure