End-to-end mBERT based Seq2seq Enhanced Dependency Parser with Linguistic Typology knowledge

Chinmay Choudhary, Colm O’riordanACL 2021

TL;DR

This paper introduces an end-to-end dependency parser that leverages a multilingual BERT model and linguistic typology knowledge to perform enhanced dependency parsing across multiple languages. The proposed model achieves state-of-the-art results while being simpler in design than previous approaches, demonstrating the effectiveness of combining large pre-trained language models with linguistic insights for complex NLP tasks.

What Problem Does This Paper Address?

Traditional dependency parsing focuses on identifying the grammatical relationships between words in a sentence. Enhanced Dependency (ED) parsing goes a step further by incorporating additional syntactic and semantic information that is often missing in standard dependency trees. This richer representation is crucial for various downstream NLP applications. The IWPT 2021 Shared Task specifically challenged participants to perform ED parsing on sentences from 17 different languages, requiring models that are not only accurate but also capable of cross-lingual transfer. Existing approaches often involve complex multi-stage pipelines, making them harder to train and implement. This paper aims to address this by proposing a unified, end-to-end system that simplifies the ED parsing process while maintaining high performance.

Methods

The core of the proposed system is an end-to-end Seq2seq mBERT-based ED parser. This model is built upon the UDify architecture, a state-of-the-art multilingual dependency parser that utilizes a shared mBERT encoder and task-specific decoders. The authors extend UDify by adding a fifth auxiliary task: a Relative Head Sequence predictor. This predictor takes the mBERT encoder's output and linguistic typology features as input and sequentially generates a tag sequence representing the enhanced dependency parse tree.

Key methodological contributions include:

  • Multitasking Learning: The model simultaneously performs five tasks: UPOS-tagging, UFeat-tagging, Lemmatization, Dependency Parsing, and Enhanced Dependency Parsing. This allows for knowledge sharing between tasks and improves overall performance.
  • Seq2seq for ED Parsing: Instead of traditional graph-based methods, the ED parse tree is represented as a relative head-position tag sequence. This sequence is then predicted by an LSTM-based Seq2seq model.
  • Linguistic Typology Integration: The model incorporates features from the World Atlas of Language Structures (WALS) database. These typology features, representing characteristics like word order and morphological complexity, are concatenated with BERT embeddings to enhance cross-lingual transfer capabilities.
  • Label Predictor: A separate mBERT-based classifier is used to predict the specific ED dependency labels for the identified relations.

Datasets

The model was trained on a large joint polyglot corpus created by concatenating all the treebanks provided for the IWPT 2021 Shared Task. Evaluation was performed on blind test corpora in eight distinct languages: Bulgarian, English, Estonian, Latvian, Lithuanian, Russian, Slovak, and Swedish.

Key Insights and Ideas

  • End-to-End Simplicity: The paper demonstrates that a unified, end-to-end approach can achieve performance comparable to more complex, multi-stage systems for ED parsing. This suggests that simplifying the pipeline doesn't necessarily sacrifice accuracy.
  • Power of Multitasking: Combining multiple linguistic tasks within a single model significantly boosts performance, especially for tasks like UFeats and Lemmatization, which benefit from shared representations.
  • Typology for Transfer: Integrating linguistic typology features from WALS proved effective in improving the model's ability to generalize across languages, highlighting the value of explicit linguistic knowledge in low-resource scenarios or for languages with different structural properties.
  • Seq2seq for Structure: Representing the ED parse tree as a sequence of relative head-position tags and predicting it with a Seq2seq model is a viable and effective alternative to traditional graph-based parsing methods for this task.

Key Results

The proposed end-to-end mBERT-based ED parser achieved state-of-the-art results on the IWPT 2021 Shared Task across the eight evaluated languages. While specific numbers vary by language and metric, the paper reports that their model performs "on par with state-of-the-art ED-parser" despite its simpler design. For instance, in the Bulgarian test set, their model achieved UPOS: 98.72, UFeats: 97.23, Lemmas: 97.25, UAS: 92.98, LAS: 89.52, and ELAS: 86.67. The authors emphasize that their model is "much simpler in design as it is an end-to-end design, and thus is much easier to train and implement."

Limitations and Future Work

The authors acknowledge that their model is simpler in design, which is presented as a strength. However, they do not explicitly detail limitations in terms of specific linguistic phenomena or language families where their model might struggle. Future work could involve exploring more sophisticated ways to integrate linguistic typology, potentially using it to guide the attention mechanisms or decoder outputs. Further investigation into the specific contributions of each auxiliary task and the impact of different typology features could also yield valuable insights. Additionally, extending this end-to-end approach to other complex NLP tasks beyond ED parsing would be a natural next step.

Summary updated

BibTeX
@inproceedings{choudhary2021end,
  author = {Chinmay Choudhary and Colm O’riordan},
  title = {{End-to-end mBERT based Seq2seq Enhanced Dependency Parser with Linguistic Typology knowledge}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
}
Figure