Using Neural Transfer Learning for Morpho-syntactic Tagging of South-Slavic Languages Tweets

Sara Meftah, Nasredine Semmar, Fatiha Sadat, Stephan RaaijmakersCOLING 2018

TL;DR

This paper tackles the challenge of morpho-syntactic tagging for South-Slavic languages on Twitter, a domain with scarce annotated data. They propose a neural network approach that leverages transfer learning from larger, out-of-domain corpora to significantly improve tagging accuracy on noisy, informal tweets.

What Problem Does This Paper Address?

Morpho-syntactic tagging, which assigns grammatical categories like gender, case, and tense to words, is a fundamental task in Natural Language Processing (NLP). However, applying this to informal social media text, especially for morphologically rich languages like Slovene, Croatian, and Serbian, is particularly difficult. These languages have complex word inflections and flexible word order, and Twitter data is characterized by non-standard orthography, abbreviations, and noise. Crucially, there's a severe lack of manually annotated Twitter data for these South-Slavic languages, making it hard to train high-performing neural models that typically require large datasets.

Methods

The authors employ a neural network architecture that combines both character-level and word-level representations. For character-level embeddings, they use a bidirectional Gated Recurrent Unit (bi-GRU) encoder to capture morphological features within tokens. Word-level embeddings are initialized using pre-trained FastText vectors to leverage semantic information. These representations are then fed into a bi-GRU sequence labeling layer.

The core innovation lies in their cross-domain transfer learning (TL) approach. They train a "parent" neural network on a large, out-of-domain corpus (standard text) for the same task. Then, they transfer the learned weights from this parent network to a "child" network, which is subsequently fine-tuned on the small, in-domain Twitter dataset. This allows the model to benefit from the linguistic knowledge learned from the larger source corpus, mitigating the data scarcity issue for the target domain. They also explore using a Conditional Random Field (CRF) layer instead of a softmax function for the final output, which is known to improve sequence labeling tasks.

Datasets

The study uses two types of datasets for each of the three South-Slavic languages (Slovene, Croatian, Serbian):

  • In-domain data: Manually annotated Twitter data (small, low-resource).
  • Out-of-domain data: Large, manually annotated raw canonical data (standard text corpora).

The paper references the MTT (Morpho-syntactic Tagging of Tweets) shared task datasets.

Key Insights and Ideas

  • Transfer learning is crucial for low-resource languages on social media: The significant performance gains demonstrate that leveraging larger, out-of-domain corpora is essential for achieving good results on noisy, data-scarce social media text.
  • Character-level embeddings are vital for handling Out-Of-Vocabulary (OOV) words: Ablation studies show that removing character-level embeddings drastically degrades performance, especially on words not seen during training, highlighting their role in capturing morphological nuances.
  • Cross-domain TL is more effective for languages with larger source corpora: The improvements from TL were more pronounced for Slovene and Croatian compared to Serbian, which had a smaller out-of-domain dataset.
  • CRFs offer a performance boost over softmax: Replacing the softmax output layer with a CRF layer further improved accuracy, indicating the benefit of modeling label dependencies.

Key Results

The paper reports significant improvements due to transfer learning:

  • Overall accuracy: Transfer learning improved accuracy by approximately 2.8% for Slovene, 2.7% for Croatian, and 0.9% for Serbian on the MTT shared task submission.
  • Ablation Study (Slovene): Removing character-level embeddings dropped overall accuracy by 7%, while removing pre-trained word embeddings dropped it by 3.4%. Performance on OOTVs (Out-Of-Training Vocabulary words) dropped significantly more when character embeddings were removed.
  • Impact of TL on POS tags (Slovene): TL led to improvements in accuracy for adjectives (+10%), nouns (+6%), and adverbs (+2%), with a slight drop for abbreviations and interjections.
  • CRF vs. Softmax: Replacing softmax with CRFs resulted in an accuracy increase of around 2.8% for Slovene, 2.5% for Croatian, and 4.9% for Serbian. The best overall accuracy achieved was 87.47% for Serbian using the CRF layer.

Limitations and Future Work

  • Limited source data for Serbian: The smaller out-of-domain dataset for Serbian limited the extent of improvement from transfer learning.
  • Future Work: The authors suggest applying their model to other morphologically rich languages, including MSA (Modern Standard Arabic) and its dialects. They also propose modeling lexical and syntactic similarities between source and target domains to further incorporate external linguistic knowledge into the neural network model.

Summary updated

BibTeX
@inproceedings{meftah2018using,
  author = {Sara Meftah and Nasredine Semmar and Fatiha Sadat and Stephan Raaijmakers},
  title = {{Using Neural Transfer Learning for Morpho-syntactic Tagging of South-Slavic Languages Tweets}},
  booktitle = {International Conference on Computational Linguistics},
  year = {2018},
}
Figure