Towards JointUD: Part-of-speech Tagging and Lemmatization using Recurrent Neural Networks

Gor Arakelyan, Karen Hambardzumyan, Hrant Khachatrian

Abstract

This paper describes our submission to CoNLL 2018 UD Shared Task. We have extended an LSTM-based neural network designed for sequence tagging to additionally generate character-level sequences. The network was jointly trained to produce lemmas, part-of-speech tags and morphological features. Sentence segmentation, tokenization and dependency parsing were handled by UDPipe 1.2 baseline. The results demonstrate the viability of the proposed multitask architecture, although its performance still remains far from state-of-the-art.

Introduction

The Universal Dependencies (UD) project aims to establish consistent grammatical annotations across diverse languages. This task encompasses several interrelated sub-problems: lemmatization (identifying base forms), part-of-speech (POS) tagging, and morphological feature prediction (e.g., gender, tense). While traditional approaches often employ sequential pipelines, we propose JointUD, a multitask neural network designed to output these labels simultaneously. By learning shared representations, the model aims to leverage the inherent dependencies between these linguistic features.

Methodology

Our system, ArmParser, utilizes a two-stage process. Initial preprocessing, including sentence segmentation and dependency parsing, is performed by UDPipe 1.2. The core contribution is a multitask neural architecture for the remaining tags.

Architecture

The model's input layer represents words by concatenating 300-dimensional FastText embeddings, an 8-feature casing vector, and a 358-dimensional character-level Bi-LSTM representation. These embeddings are processed by three layers of LSTMs (150 units each) to extract contextual features.

The output stage consists of two main components:

  1. Tagging Classifiers: 22 separate softmax classifiers predict POS and various morphological features (e.g., Number, Gender, Tense).
  2. Lemma Decoder: A GRU-based decoder generates lemmas character-by-character. Crucially, the decoder receives the word's contextual representation at every step, along with character embeddings and a reverse positional encoding (counting characters from the end of the word) to better handle suffix-based morphology.
Distribution of morphological tags and values predicted by the multitask classifiers.

Training

To balance the varying difficulty of tasks, we employed a weighted loss function. Specifically, the POS tagging loss weight was reduced ($\lambda_p = 0.2$) to prevent it from dominating the training process and to allow the model to focus on the more complex lemma generation task.

Experimental Results

We evaluated the model on various treebanks from the CoNLL 2018 UD Shared Task, including English, French, Finnish, and Swedish, with Arabic and Korean used to test cross-lingual generalization.

Performance comparison of the proposed model (Our) against UDPipe and the task winner for POS, UFeat, and Lemma metrics.

The results indicate that while the multitask architecture is viable, it generally performs below the UDPipe 1.2 baseline. For instance, on the English EWT dataset, our model achieved a Lemma score of 91.51% and a POS score of 90.47%, compared to UDPipe's 95.92% and 93.61%, respectively.

A significant challenge was observed with non-Indo-European languages. The model struggled with Arabic (7.42% Lemma score) and Korean (12.87% Lemma score), suggesting the current decoder logic doesn't generalize well to different morphological systems.

Discussion and Future Work

Our findings highlight the importance of persistent context in the lemma decoder; providing word-level context at every generation step was vital for distinguishing between base characters and grammatical suffixes. However, the current architecture's reliance on reverse positional encoding limits its effectiveness for diverse morphological systems.

Future work will focus on:

  • Developing more flexible decoder architectures to improve generalization across diverse language families.
  • Optimizing computational efficiency, as the lemma generation component currently increases training time significantly.
  • Integrating dependency parsing directly into the multitask network to achieve a fully unified JointUD model.

Summary updated

Abstract

This paper describes our submission to CoNLL 2018 UD Shared Task. We have extended an LSTM-based neural network designed for sequence tagging to additionally generate character-level sequences. The network was jointly trained to produce lemmas, part-of-speech tags and morphological features. Sentence segmentation, tokenization and dependency parsing were handled by UDPipe 1.2 baseline. The results demonstrate the viability of the proposed multitask architecture, although its performance still remains far from state-of-the-art.
BibTeX
@inproceedings{arakelyan2018towards,
  author = {Gor Arakelyan and Karen Hambardzumyan and Hrant Khachatrian},
  title = {{Towards JointUD: Part-of-speech Tagging and Lemmatization using Recurrent Neural Networks}},
  booktitle = {CoNLL Shared Task},
  year = {2018},
  url = {https://openreview.net/forum?id=-zo3MmxYjf},
}
Figure