What does Neural Bring? Analysing Improvements in Morphosyntactic Annotation and Lemmatisation of Slovenian, Croatian and Serbian

Nikola Ljubešić, Kaja DobrovoljcACL 2019

TL;DR

This paper investigates the impact of neural network models on morphosyntactic annotation and lemmatisation for three South Slavic languages. They found that neural approaches significantly improve morphosyntactic annotation, especially for complex linguistic phenomena, but offer no gains in lemmatisation compared to traditional methods, highlighting the continued importance of lexicons for this task.

What Problem Does This Paper Address?

Morphosyntactic annotation (tagging words with their part-of-speech and grammatical features) and lemmatisation (reducing words to their base or dictionary form) are fundamental tasks in Natural Language Processing (NLP), particularly for morphologically rich languages like Slovenian, Croatian, and Serbian. While traditional methods using carefully engineered features and lexicons have achieved state-of-the-art results, the NLP field has seen a surge in neural network approaches. This paper aims to quantify the improvements, if any, that modern neural pipelines bring to these specific tasks for these under-resourced Slavic languages, comparing them against a strong traditional baseline.

Methods

The authors compare two main approaches:

  • Traditional: reldi-tagger, a CRF-based tagger that leverages hand-crafted features and an inflectional lexicon. This was previously the state-of-the-art for these languages.
  • Neural: stanfordnlp, a neural network pipeline that participated in the CoNLL 2018 shared task and is known for its strong performance in morphosyntactic and dependency parsing.

The evaluation uses a modified CoNLL 2018 shared task script. For morphosyntactic annotation, they predict Multi-Lingual Tagset (MLT) descriptions. For lemmatisation, they explore several scenarios: using gold morphosyntax, using predicted morphosyntax from the same tool, and a novel approach where stanfordnlp is augmented with external inflectional lexicons (stanfordnlp+lex). They also experiment with different word embeddings (Brown clusters, CoNLL 2017 word2vec, and CLARIN.SI embeddings trained with word2vec or fastText).

Datasets

The experiments are conducted on three South Slavic languages: Slovenian, Croatian, and Serbian. The datasets used are:

  • Slovenian: ssj500k (approx. 500k tokens)
  • Croatian: hr500k (approx. 500k tokens)
  • Serbian: SE-Times.SR (approx. 87k tokens)

These datasets are part of the babushka-bench benchmarking platform, which provides train, development, and test splits. The paper also utilizes inflectional lexicons for each language: Sloleks (Slovenian), hrLex (Croatian), and srLex (Serbian).

Key Insights and Ideas

  • Neural networks excel at complex annotation: The neural stanfordnlp significantly outperforms the traditional reldi-tagger on morphosyntactic annotation, particularly for categories requiring semantic knowledge or modelling long-range dependencies.
  • Lexicons remain king for lemmatisation: Despite advancements in neural architectures, stanfordnlp does not surpass reldi-tagger in lemmatisation. This is attributed to the task's heavy reliance on lookup in large, external inflectional lexicons.
  • Augmenting neural with lexicons bridges the gap: By adding external inflectional lexicons to stanfordnlp (stanfordnlp+lex), the performance gap in lemmatisation is almost entirely closed, demonstrating the power of combining neural context with lexical knowledge.
  • Embeddings matter, but not always drastically: While different embedding choices show minor variations, CLARIN.SI embeddings generally perform better. FastText embeddings show a slight advantage when training data is scarce (as with Serbian).
  • Long-range dependencies are a neural strength: The paper highlights that stanfordnlp is better at resolving ambiguities that depend on context further away in the sentence, such as distinguishing between noun cases or between adjectives and adverbs.

Key Results

  • Morphosyntactic Annotation: stanfordnlp achieved a relative error reduction of 43% for Slovenian, 27% for Croatian, and 40% for Serbian compared to reldi-tagger. Table 1 shows stanfordnlp achieving F1 scores in the mid-90s across languages and embedding types, generally outperforming reldi-tagger's scores in the low-to-mid 90s.
  • Lemmatisation: When using gold morphosyntax, reldi-tagger achieved higher F1 scores (around 98-99%) than stanfordnlp (around 95-97%). However, with the stanfordnlp+lex modification, the performance became nearly identical to reldi-tagger (around 97-98%). Table 3 details these results.

Limitations and Future Work

  • Lemmatisation dependence on lexicons: The paper explicitly states that lemmatisation performance is heavily dependent on lexicon lookup, which is a limitation for purely neural approaches without such resources.
  • stanfordnlp's bug: The authors note that stanfordnlp had an "unfortunate bug" that, if fixed, would have placed it among the top-three systems in the CoNLL 2018 shared task, suggesting potential for even better performance.
  • Future work: The paper implicitly suggests further research into integrating external lexicons more effectively into neural models for lemmatisation. The development of the babushka-bench platform itself is a contribution, enabling future research and benchmarking for these languages.

Summary updated

BibTeX
@inproceedings{ljubesic2019what,
  author = {Nikola Ljubešić and Kaja Dobrovoljc},
  title = {{What does Neural Bring? Analysing Improvements in Morphosyntactic Annotation and Lemmatisation of Slovenian, Croatian and Serbian}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2019},
}
Figure