Towards JointUD: Part-of-speech Tagging and Lemmatization using Recurrent Neural Networks

Gor Arakelyan, Karen Hambardzumyan, Hrant Khachatrian

Here is a concise, technical summary of the paper.

TL;DR

The authors present an early step toward JointUD, an end-to-end multitask neural architecture that jointly predicts part-of-speech (POS) tags, morphological features, and character-level lemmas from tokenized text. Submitted to the CoNLL 2018 Universal Dependencies (UD) Shared Task, the system augments an LSTM sequence tagger with parallel, character-level GRU decoders to generate lemmas alongside token-level classification heads.

What Problem Does This Paper Address?

In the Universal Dependencies (UD) framework, natural language sentences are annotated with token-level POS tags, fine-grained morphological features, lemmas (base dictionary forms), and syntactic dependency trees. Traditionally, pipelines address these tasks piecemeal or sequentially (e.g., POS tagging first, followed by rule-based or separate statistical lemmatizers). Joint architectures that concurrently predict both token-level morphosyntactic labels and character-level lemma sequences had rarely been explored together. The authors investigate whether a single recurrent neural network can jointly learn POS tagging, morphological analysis, and character-by-character lemmatization.

Methods

The model pipeline uses UDPipe 1.2 for segmentation, tokenization, and dependency parsing, while a custom neural network replaces UDPipe's lemma and morphosyntax outputs:

  1. Embedding Layer: Each word is mapped to a 358-dimensional vector concatenating:
  2. 300-d pretrained FastText embeddings.
  3. 8-d one-hot casing features (e.g., all lowercase, initial uppercase, numeric).
  4. 50-d character-level representation (final states of a bidirectional LSTM applied to 30-d character embeddings).
  5. Feature Extractor: A 3-layer LSTM with 150-dimensional hidden states and 50% dropout between layers creates context-aware word representations ($h_i^3$).
  6. Classification Heads: Softmax linear layers over $h_i^3$ predict the universal POS tag and up to 21 language-specific morphological features (evaluated with 22 cross-entropy losses).
  7. Lemma Decoder (Novel Component): A shared, parallel GRU decoder generates characters for each word's lemma. At step $j$ for word $i$, the GRU inputs:
  8. Word context vector $h_i^3$ (fed at every step to prevent forgetting).
  9. 30-d embedding of the corresponding input word character $c_i^j$.
  10. A reverse positional embedding $\pi_i^j = \text{Emb}^5(n_i - j + 1)$, marking characters remaining until the end of the input word.
  11. The embedding/one-hot of the previously generated lemma character.
  12. Multitask Loss: A weighted sum $L = \lambda_l L_l + \lambda_p L_p + \sum_{k=1}^{21} \lambda_{f_k} L_{f_k}$, where empirical tuning set $\lambda_p = 0.2$ and $\lambda_l = \lambda_{f_k} = 1.0$ to prevent POS loss from dominating.

Datasets

The model was evaluated on the Universal Dependencies (UD) v2.2 benchmarks:

  • Primary experiments used 9 treebanks: English (EWT, GUM, LinES), French (Spoken, Sequoia), Finnish (TDT, FTB), and Swedish (LinES, Talbanken).
  • Additional local post-competition experiments tested two non-Indo-European treebanks: Arabic (PADT) and Korean (GSD).
  • No new datasets were introduced.

Key Insights and Ideas

  • Context Injection Prevents Premature Truncation: Early decoder prototypes generated early end-of-token predictions (e.g., stripping the plural suffix -s led to truncating English words at internal s characters). Injecting the top-level word context vector ($h_i^3$) into the GRU at every step rather than just at initialization successfully resolved this.
  • Reverse Positional Encodings: Supplying the number of characters remaining until the end of the token significantly improved lemma generation for suffix-heavy morphology (such as English inflections).
  • Loss Scaling Matters: Without down-weighting the POS loss ($\lambda_p = 0.2$), the model heavily prioritized POS classification at the expense of lemma generation quality.

Key Results

  • Overall Performance: The system did not outperform the baseline UDPipe 1.2 across most metrics. On English EWT, local evaluation showed:
  • POS: 90.47% vs. UDPipe's 93.61% (shared task winner: 95.94%).
  • Morphological Features (UFeat): 93.95% vs. UDPipe's 94.60% (winner: 96.03%).
  • Lemma: 91.51% vs. UDPipe's 95.92% (winner: 97.23%).
  • MLAS / BLEX: 62.12 / 66.35 vs. UDPipe's 68.27 / 70.53.
  • TIRA Shared Task Bug: An input preprocessing bug on the competition platform duplicated newlines, breaking sentence segmentation and artificially dropping submitted scores (e.g., English EWT BLEX scored 60.73 on TIRA vs. 66.35 locally).
  • Non-Indo-European Degradation: The lemmatization architecture failed severely on Arabic PADT (7.42% Lemma score vs. UDPipe’s 87.42%) and Korean GSD (12.87% vs. 87.03%), demonstrating a lack of generalization to root-and-pattern or agglutinative morphologies.

Limitations and Future Work

  • Poor Cross-Lingual Generalization: The reverse-positional character decoder relies heavily on suffix-based transformations and fails drastically on non-Indo-European languages.
  • Compute and Speed: Adding parallel character-level decoders slowed training down by more than $3\times$.
  • Dependency Parsing Not Integrated: True joint parsing was not achieved; dependency parsing was offloaded entirely to UDPipe 1.2.
  • Future Directions: The authors highlight hyperparameter optimization, advanced regularization (e.g., recurrent dropout), network redesign for non-concatenative morphology, and extending the model to incorporate full dependency parsing into a single unified network.

Summary updated

Abstract

This paper describes our submission to CoNLL 2018 UD Shared Task. We have extended an LSTM-based neural network designed for sequence tagging to additionally generate character-level sequences. The network was jointly trained to produce lemmas, part-of-speech tags and morphological features. Sentence segmentation, tokenization and dependency parsing were handled by UDPipe 1.2 baseline. The results demonstrate the viability of the proposed multitask architecture, although its performance still remains far from state-of-the-art.
BibTeX
@inproceedings{arakelyan2018towards,
  author = {Gor Arakelyan and Karen Hambardzumyan and Hrant Khachatrian},
  title = {{Towards JointUD: Part-of-speech Tagging and Lemmatization using Recurrent Neural Networks}},
  booktitle = {CoNLL Shared Task},
  year = {2018},
  url = {https://openreview.net/forum?id=-zo3MmxYjf},
}

Loading related content...

Figure