Capturing Dependency Syntax with "Deep" Sequential Models

Yoav GoldbergDepLing 2017

Abstract

Traditionally, linguistic theory posits that language is structured hierarchically through tree-like dependencies. In contrast, deep learning models such as Recurrent Neural Networks (RNNs) are inherently sequential, processing information in a linear chain. This paper explores the conceptual tension between these two paradigms, investigating whether sequential models can truly capture the nested, long-distance relationships required for complex grammar. By examining subject-verb agreement and the performance of Bi-directional RNNs in dependency parsing, this research demonstrates that sequential models possess a remarkable capacity for modeling hierarchical syntax without explicit tree-structured biases.

Introduction

A fundamental question in computational linguistics is whether models that process words one-by-one can understand the underlying structural relationships of a language. For decades, the consensus was that capturing syntax required explicit hierarchical architectures. However, the rise of "deep" sequential models has challenged this binary. This paper, based on a keynote by Yoav Goldberg, evaluates the syntactic competence of RNNs, specifically their ability to handle dependency syntax—the grammatical relations between words regardless of their linear distance.

The Sequential vs. Hierarchical Debate

In dependency syntax, the relationship between a head (e.g., a verb) and its dependent (e.g., a subject) is the primary unit of structure. Because these related words can be separated by numerous "intervener" words, a simple Markovian model—which only considers the immediate previous word—is insufficient. The central problem addressed here is whether RNNs are merely "stochastic parrots" of local patterns or if they maintain a hidden state robust enough to track these long-distance dependencies across a sentence.

Methodology: Probing Syntactic Knowledge

To test the limits of sequential architectures, the research focuses on two primary models:

  1. Standard RNNs: These are non-Markovian models that maintain a persistent hidden state. They were tested on the specific task of subject-verb agreement to see if the hidden state could "remember" the number (singular or plural) of a subject across multiple intervening words.
  2. Bi-directional RNNs (Bi-RNNs): This extension processes sequences in both forward and backward directions. By concatenating the representations from both passes, the model gains a "global" view of the sentence, providing context from both the preceding and succeeding words for any given token.

These representations were then utilized as the core features for building dependency parsers (notably in collaboration with Kipperwasser). These parsers were trained on naturally occurring text across a variety of languages to identify the formal grammatical structure of sentences.

Key Insights and Syntactic Competence

The empirical evidence suggests that sequential models are surprisingly syntactically competent. Despite the lack of an explicit hierarchical bias, RNNs can learn the nuances of subject-verb agreement through indirect supervision. They do not need to be explicitly taught grammar rules; instead, they extract these patterns from the statistical regularities of the language itself.

Furthermore, the success of Bi-directional RNNs indicates that bi-directionality is a powerful proxy for structural understanding. By looking at the entire sentence at once, the Bi-RNN effectively bridges the gap between linear processing and hierarchical representation. This challenged the long-held belief that a model must be "tree-shaped" to understand a "tree-shaped" language.

Results and Impact

The findings presented in this work led to several significant conclusions:

  • Robustness to Interveners: RNNs successfully maintain syntactic dependencies even when complex clauses or multiple words intervene between the subject and the verb.
  • State-of-the-Art Performance: The transition to Bi-RNN-based features allowed dependency parsers to reach new levels of accuracy, setting state-of-the-art benchmarks at the time of the research (2017).
  • Cross-Linguistic Generalization: The ability of these models to capture syntax was not a quirk of English grammar but was validated across multiple languages with diverse syntactic rules, proving the universality of the approach.

Conclusion

The research demonstrates that the "linear" nature of RNNs is not a barrier to capturing "deep" syntactic dependencies. By leveraging hidden states and bi-directional context, sequential models can effectively internalize the hierarchical structures of human language. This shift in understanding has paved the way for modern NLP architectures, proving that sophisticated linguistic representations can emerge from sequential processing given sufficient depth and data.

Summary updated

BibTeX
@inproceedings{goldberg2017capturing,
  author = {Yoav Goldberg},
  title = {{Capturing Dependency Syntax with "Deep" Sequential Models}},
  booktitle = {DepLing},
  year = {2017},
}
Figure