What Should/Do/Can LSTMs Learn When Parsing Auxiliary Verb Constructions?

Miryam de Lhoneux, Sara Stymne, Joakim Nivre

Abstract

There is a growing interest in investigating what neural NLP models learn about language. A prominent open question is whether it is necessary to model hierarchical structure. We present a linguistic investigation of a neural parser, looking at transitivity and agreement information of auxiliary verb constructions (AVCs) in comparison to finite main verbs (FMVs). This comparison is motivated by dependency grammar, specifically the work of Tesnière (1959), where AVCs and FMVs are both instances of a "nucleus"—the basic unit of syntax. An AVC is a dissociated nucleus consisting of at least two words, while an FMV is its non-dissociated counterpart. We suggest that the representation of AVCs and FMVs should capture similar information. Using diagnostic classifiers to probe vectors learned by a transition-based neural parser in four typologically different languages, we find that parsers learn different information about AVCs and FMVs with sequential models (BiLSTMs) but similar information when a recursive layer is used.

Introduction: The Nucleus and the Parser

In dependency grammar, the nucleus is considered the fundamental unit of syntax. A simple verb like "ran" (a Finite Main Verb, or FMV) is a single-word nucleus. In contrast, complex phrases like "could have run" (an Auxiliary Verb Construction, or AVC) form what is known as a dissociated nucleus. Despite being spread across multiple words, an AVC functions as a single unit in terms of its relationship with other sentence elements.

The concept of a dissociated nucleus (AVC) where multiple words function as a single syntactic unit.

Modern neural parsers, typically based on BiLSTMs, are highly effective at predicting tree structures. However, a critical question remains: do these models actually learn the underlying linguistic reality that an AVC should represent the same functional information (such as agreement or transitivity) as an FMV? If a parser treats "could have run" fundamentally differently than "ran," it may be failing to capture a key inductive bias of human language. The authors specifically ask: Can LSTMs learn this "dissociated nucleus" concept, and does adding hierarchical (recursive) structure help?

Methodology

The researchers employed a transition-based neural parser (UUParser) across four typologically diverse languages: Catalan, Croatian, Dutch, and Finnish. The goal was to determine if the internal representations (vectors) of these verbs contained specific linguistic features.

Probing for Information

The study used diagnostic classifiers (probes) to extract "frozen" vector representations from the trained parser. A simple Multi-Layer Perceptron (MLP) was then trained to identify two key features:

  1. Agreement: Does the vector encode the person and number of the subject?
  2. Transitivity: Does the vector encode whether the verb takes a direct object?

Architectural and Representational Variations

The study compared a standard sequential BiLSTM parser against one augmented with a recursive layer designed to compose subtree representations. Furthermore, they tested two different annotation styles:

  • Universal Dependencies (UD): Where the main verb is the head of the construction.
  • Mel’cuk-style (MS): Where the auxiliary verb acts as the head.
Comparison of dependency representations for finite main verbs (top) and auxiliary verb constructions in UD (middle) and Mel’cuk-style (bottom) formats.

Key Insights: The Agreement Gap

The research revealed a significant "Information Flow Mismatch" in standard BiLSTM parsers using UD annotations. While the models successfully learned transitivity for both FMVs and AVCs, they struggled with agreement in complex constructions.

The reason is structural: in an AVC, transitivity is a property of the main verb, which the BiLSTM can easily detect in context. However, agreement features (e.g., "3rd person singular") are typically carried by the auxiliary verb. In a purely sequential model, this information often fails to propagate into the main verb's vector representation.

The Impact of Recursion

Adding a recursive layer—which explicitly merges the auxiliary and main verb into a single subtree vector—effectively bridges this gap. This hierarchical composition allows the "dissociated" parts of the nucleus to share information, resulting in an AVC representation that closely mirrors that of a simple FMV.

Results and Discussion

The quantitative results highlight the effectiveness of recursive layers:

  • Agreement Identification: For FMVs, token vectors were significantly better (~13.7%) than the baseline at identifying agreement. In standard BiLSTMs, this performance dropped to near zero for AVCs. With the recursive layer, AVC vectors captured agreement at levels comparable to simple verbs.
  • Transitivity Consistency: Both sequential and recursive models were successful at capturing transitivity across all verb types, as this information is more locally available.

Interestingly, while the recursive layer produced more linguistically "correct" internal representations, it did not lead to a significant improvement in overall parsing accuracy (Labeled Attachment Score, or LAS). This suggests that current evaluation metrics may not fully capture the qualitative improvements in how models represent language.

Conclusion and Future Directions

The study concludes that while LSTMs can be pushed to learn hierarchical concepts like the dissociated nucleus, they require specific architectural biases—like recursive layers—to do so effectively for features like agreement.

Future research will explore other dissociated structures (such as adposition-noun combinations) and investigate whether these more robust recursive representations provide benefits in downstream tasks like machine translation or natural language inference, where deep semantic understanding is paramount.

Summary updated

Abstract

There is a growing interest in investigating what neural NLP models learn about language. A prominent open question is the question of whether or not it is necessary to model hierarchical structure. We present a linguistic investigation of a neural parser adding insights to this question. We look at transitivity and agreement information of auxiliary verb constructions (AVCs) in comparison to finite main verbs (FMVs). This comparison is motivated by theoretical work in dependency grammar and in particular the work of Tesni\`ere (1959) where AVCs and FMVs are both instances of a nucleus, the basic unit of syntax. An AVC is a dissociated nucleus, it consists of at least two words, and an FMV is its non-dissociated counterpart, consisting of exactly one word. We suggest that the representation of AVCs and FMVs should capture similar information. We use diagnostic classifiers to probe agreement and transitivity information in vectors learned by a transition-based neural parser in four typologically different languages. We find that the parser learns different information about AVCs and FMVs if only sequential models (BiLSTMs) are used in the architecture but similar information when a recursive layer is used. We find explanations for why this is the case by looking closely at how information is learned in the network and looking at what happens with different dependency representations of AVCs. We conclude that there may be benefits to using a recursive layer in dependency parsing and that we have not yet found the best way to integrate it in our parsers.
BibTeX
@inproceedings{lhoneux2020what,
  author = {Miryam de Lhoneux and Sara Stymne and Joakim Nivre},
  title = {{What Should/Do/Can LSTMs Learn When Parsing Auxiliary Verb Constructions?}},
  booktitle = {Comput. Linguistics},
  year = {2020},
  url = {https://openreview.net/forum?id=EJbm_-D7YXO},
}
Figure