What Should/Do/Can LSTMs Learn When Parsing Auxiliary Verb Constructions?

Miryam de Lhoneux, Sara Stymne, Joakim Nivre

TL;DR

This paper investigates whether neural dependency parsers truly "understand" complex verb phrases (like "has been eating") as single functional units, similar to simple verbs ("eats"). The authors find that while standard BiLSTM-based parsers struggle to capture agreement information for these complex constructions, adding a recursive composition layer allows the model to represent them much more like their single-word counterparts.

What Problem Does This Paper Address?

In linguistics, specifically dependency grammar, there is a concept called a nucleus—the basic unit of syntax. A simple finite main verb (FMV) like "ran" is a single-word nucleus. However, an auxiliary verb construction (AVC) like "could have run" is a dissociated nucleus: multiple words that function together as one unit.

Modern neural parsers, typically built on BiLSTMs, are incredibly accurate at predicting the tree structure of a sentence. However, it’s unclear if they actually learn the underlying linguistic reality that an AVC should represent the same kind of information (like person/number agreement or transitivity) as an FMV. If a parser treats "could have run" fundamentally differently than "ran," it might be missing a key inductive bias of human language. The authors specifically ask: Can LSTMs learn this "dissociated nucleus" concept, and does adding hierarchical (recursive) structure help?

Methods

The researchers used a transition-based neural parser (UUParser) and applied diagnostic classifiers (also known as "probes").

  1. Training: They trained the parser on four different languages.
  2. Probing: They extracted "frozen" vector representations (type, character, and token-level) from the trained parser.
  3. Classification: They trained a separate, simple MLP to see if these vectors contained information about two specific tasks:
  4. Agreement: Does the vector know the person and number of the subject?
  5. Transitivity: Does the vector know if the verb has a direct object?
  6. Architectural Comparison: They compared a standard BiLSTM parser (sequential) against one augmented with a recursive layer that composes subtree representations. They also tested two different annotation styles: Universal Dependencies (UD), where the main verb is the head, and a "Mel’cuk-style" (MS) representation, where the auxiliary is the head.

Datasets

The study utilized Universal Dependencies (UD) version 2.2 treebanks. To ensure the results weren't language-specific, they selected four typologically diverse languages with high-quality annotations and a sufficient number of auxiliary constructions:

  • Catalan (AnCora)
  • Croatian (SET)
  • Dutch (Alpino)
  • Finnish (TDT)

Key Insights and Ideas

  • Information Flow Mismatch: In standard BiLSTM parsers using UD (where the main verb is the head), the model successfully learns transitivity for complex verb phrases but fails to capture agreement.
  • The "Why": Transitivity is a property of the main verb, which the BiLSTM can easily "see" in context. However, agreement features (like "3rd person singular") are usually physically located on the auxiliary verb. In a sequential model, this information doesn't always propagate effectively into the main verb's vector representation.
  • Recursion as a Bridge: Adding a recursive layer—which explicitly merges the auxiliary and the main verb into a single subtree vector—solves this. It allows the "dissociated" parts of the nucleus to share information, making the final representation of an AVC look remarkably similar to a simple FMV.
  • The Representation Paradox: Interestingly, while the recursive layer makes the internal representations more linguistically "correct" (capturing the dissociated nucleus), it does not improve overall parsing accuracy (LAS). This suggests that current evaluation metrics might not capture the qualitative improvements in how models represent language.

Key Results

  • Agreement Gap: For simple verbs (FMVs), token vectors were ~13.7% better than the baseline at identifying agreement. For complex constructions (AVCs) in a standard BiLSTM, this dropped to near zero—the model essentially "forgot" the agreement info for the main verb.
  • Recursive Success: When the recursive layer was added, the AVC vectors captured agreement information at levels comparable to simple verbs, effectively learning the concept of the dissociated nucleus.
  • Transitivity Consistency: Both sequential and recursive models were successful at capturing transitivity for both simple and complex verbs, as this information is more readily available in the local context.

Limitations and Future Work

The biggest limitation is the "Accuracy Gap": if recursive layers provide more linguistically sound representations, why don't they improve parsing scores? The authors suggest we haven't yet found the optimal way to integrate these recursive layers into modern architectures.

Future work includes:

  • Investigating other "dissociated" structures, such as adposition-noun combinations.
  • Exploring whether these "better" recursive representations actually help in downstream tasks (like translation or natural language inference) where understanding the relationship between subjects and complex verbs is crucial.
  • Refining the recursive composition function to see if different architectures (like Transformers) handle these dissociated nuclei differently.

Summary updated

Abstract

There is a growing interest in investigating what neural NLP models learn about language. A prominent open question is the question of whether or not it is necessary to model hierarchical structure. We present a linguistic investigation of a neural parser adding insights to this question. We look at transitivity and agreement information of auxiliary verb constructions (AVCs) in comparison to finite main verbs (FMVs). This comparison is motivated by theoretical work in dependency grammar and in particular the work of Tesnière (1959), where AVCs and FMVs are both instances of a nucleus, the basic unit of syntax. An AVC is a dissociated nucleus; it consists of at least two words, and an FMV is its non-dissociated counterpart, consisting of exactly one word. We suggest that the representation of AVCs and FMVs should capture similar information. We use diagnostic classifiers to probe agreement and transitivity information in vectors learned by a transition-based neural parser in four typologically different languages. We find that the parser learns different information about AVCs and FMVs if only sequential models (BiLSTMs) are used in the architecture but similar information when a recursive layer is used. We find explanations for why this is the case by looking closely at how information is learned in the network and looking at what happens with different dependency representations of AVCs. We conclude that there may be benefits to using a recursive layer in dependency parsing and that we have not yet found the best way to integrate it in our parsers.
BibTeX
@inproceedings{lhoneux2020what,
  author = {Miryam de Lhoneux and Sara Stymne and Joakim Nivre},
  title = {{What Should/Do/Can LSTMs Learn When Parsing Auxiliary Verb Constructions?}},
  booktitle = {Comput. Linguistics},
  year = {2020},
  url = {https://openreview.net/forum?id=EJbm_-D7YXO},
}

Loading related content...

Figure