Explicitly modeling case improves neural dependency parsing

Clara Vania, Adam LopezEMNLP 2018

TL;DR

Neural dependency parsers, despite learning word representations from characters, still struggle to fully capture morphological case information. Explicitly providing this case information, especially through a pipeline approach, significantly boosts parsing accuracy, suggesting that current character-level models don't implicitly learn all crucial linguistic features.

What Problem Does This Paper Address?

Modern neural dependency parsers often learn word representations by processing characters, which intuitively should allow them to infer morphological information. However, a key question remains: do these character-level models truly understand morphology, or do they just learn superficial patterns? This paper specifically investigates whether these models can effectively learn and utilize morphological case, a crucial feature for parsing in morphologically rich languages (MRLs). The authors aim to determine if explicit morphological information is still necessary for achieving optimal parsing performance in MRLs, even with advanced neural architectures.

Methods

The core of the paper involves experimenting with a standard neural graph-based dependency parser, similar to those proposed by Kiperwasser and Goldberg (2016) and Zhang et al. (2017). The authors treat the parser itself as a black box and focus on manipulating its input representations. They explore several input configurations:

  • Word embeddings: Standard input.
  • Characters: Input processed through a character-level bidirectional LSTM (bi-LSTM) to generate word representations.
  • Characters + predicted case: The character-level representation is augmented with predicted morphological case labels.
  • Characters + gold case: The character-level representation is augmented with the true, human-annotated morphological case labels.
  • Characters + full analysis: The character-level representation is augmented with all available morphological features.

They also investigate two methods for incorporating case information:

  • Multi-task learning (MTL): Case prediction is trained as an auxiliary task alongside dependency parsing, using the lower-layer bi-LSTM states.
  • Pipeline model: A separate morphological tagger is trained to predict case, and these predictions are then fed as input to the dependency parser.

Datasets

The experiments were conducted on three fusional languages with extensive case systems:

  • Czech: 68.5K training sentences.
  • German: 14.1K training sentences.
  • Russian: 48.8K training sentences.

The paper does not mention the introduction of new datasets; it utilizes existing datasets for these languages.

Key Insights and Ideas

  • Character-level models aren't a silver bullet for morphology: Despite their ability to learn from characters, these models don't automatically infer all necessary morphological information, specifically case. Explicitly providing case information still yields significant improvements.
  • Gold case is powerful: Using the true morphological case labels ("gold case") leads to substantial parsing gains, indicating the importance of this feature.
  • Predicted case is surprisingly helpful: Unlike older parsers where predicted features often hurt performance, here, predicted case improves parsing accuracy, even if not as much as gold case. This suggests that even imperfect morphological information can be beneficial.
  • Pipeline beats multi-task for parsing: A pipeline approach, where case is predicted first and then fed to the parser, outperforms multi-task learning for dependency parsing accuracy. This is counter-intuitive, as MTL is often expected to allow for better integration of features.
  • MTL's parsing deficit: Interestingly, the MTL model achieved better case tagging accuracy than the pipeline model, yet performed worse in dependency parsing. This suggests that while MTL can learn to represent case, it might not effectively use that representation for the parsing task.

Key Results

  • Oracle (gold case) improvements: Adding gold case to character-level models improved parsing accuracy (Label Attachment Score - LAS) by up to 2.4 LAS points on the test set for German. This also explained all the gains from a full morphological analysis.
  • Predicted case gains: Using predicted case also improved LAS scores across all languages compared to character-only models.
  • Pipeline vs. MTL: The pipeline model using predicted case generally achieved higher LAS scores than the MTL models. For example, in Czech, the pipeline model with predicted case achieved 92.2% LAS (dev) and 91.8% LAS (test), while the character (multi-task) model achieved 91.6% LAS (dev) and 91.0% LAS (test).

Limitations and Future Work

  • Limitations: The authors acknowledge that their taggers for predicting case used gold part-of-speech (POS) tags, which might have contributed to the success of predicted case. They also note that the MTL model's inability to effectively use case for parsing is an interesting, albeit unexplained, phenomenon.
  • Future Work: The paper implicitly suggests that further research into how to best integrate predicted morphological features into neural parsers is warranted. Exploring more sophisticated ways to combine multi-task learning or developing better pipeline architectures could be fruitful directions. The findings also open questions about what other linguistic features character-level models might be missing.

Summary updated

BibTeX
@inproceedings{vania2018explicitly,
  author = {Clara Vania and Adam Lopez},
  title = {{Explicitly modeling case improves neural dependency parsing}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2018},
}
Figure