Here is a concise, technical summary of the paper.
TL;DR
The authors present an early step toward JointUD, an end-to-end multitask neural architecture that jointly predicts part-of-speech (POS) tags, morphological features, and character-level lemmas from tokenized text. Submitted to the CoNLL 2018 Universal Dependencies (UD) Shared Task, the system augments an LSTM sequence tagger with parallel, character-level GRU decoders to generate lemmas alongside token-level classification heads.
What Problem Does This Paper Address?
In the Universal Dependencies (UD) framework, natural language sentences are annotated with token-level POS tags, fine-grained morphological features, lemmas (base dictionary forms), and syntactic dependency trees. Traditionally, pipelines address these tasks piecemeal or sequentially (e.g., POS tagging first, followed by rule-based or separate statistical lemmatizers). Joint architectures that concurrently predict both token-level morphosyntactic labels and character-level lemma sequences had rarely been explored together. The authors investigate whether a single recurrent neural network can jointly learn POS tagging, morphological analysis, and character-by-character lemmatization.
Methods
The model pipeline uses UDPipe 1.2 for segmentation, tokenization, and dependency parsing, while a custom neural network replaces UDPipe's lemma and morphosyntax outputs:
- Embedding Layer: Each word is mapped to a 358-dimensional vector concatenating:
- 300-d pretrained FastText embeddings.
- 8-d one-hot casing features (e.g., all lowercase, initial uppercase, numeric).
- 50-d character-level representation (final states of a bidirectional LSTM applied to 30-d character embeddings).
- Feature Extractor: A 3-layer LSTM with 150-dimensional hidden states and 50% dropout between layers creates context-aware word representations ($h_i^3$).
- Classification Heads: Softmax linear layers over $h_i^3$ predict the universal POS tag and up to 21 language-specific morphological features (evaluated with 22 cross-entropy losses).
- Lemma Decoder (Novel Component): A shared, parallel GRU decoder generates characters for each word's lemma. At step $j$ for word $i$, the GRU inputs:
- Word context vector $h_i^3$ (fed at every step to prevent forgetting).
- 30-d embedding of the corresponding input word character $c_i^j$.
- A reverse positional embedding $\pi_i^j = \text{Emb}^5(n_i - j + 1)$, marking characters remaining until the end of the input word.
- The embedding/one-hot of the previously generated lemma character.
- Multitask Loss: A weighted sum $L = \lambda_l L_l + \lambda_p L_p + \sum_{k=1}^{21} \lambda_{f_k} L_{f_k}$, where empirical tuning set $\lambda_p = 0.2$ and $\lambda_l = \lambda_{f_k} = 1.0$ to prevent POS loss from dominating.
Datasets
The model was evaluated on the Universal Dependencies (UD) v2.2 benchmarks:
- Primary experiments used 9 treebanks: English (EWT, GUM, LinES), French (Spoken, Sequoia), Finnish (TDT, FTB), and Swedish (LinES, Talbanken).
- Additional local post-competition experiments tested two non-Indo-European treebanks: Arabic (PADT) and Korean (GSD).
- No new datasets were introduced.
Key Insights and Ideas
- Context Injection Prevents Premature Truncation: Early decoder prototypes generated early
end-of-tokenpredictions (e.g., stripping the plural suffix -s led to truncating English words at internal s characters). Injecting the top-level word context vector ($h_i^3$) into the GRU at every step rather than just at initialization successfully resolved this. - Reverse Positional Encodings: Supplying the number of characters remaining until the end of the token significantly improved lemma generation for suffix-heavy morphology (such as English inflections).
- Loss Scaling Matters: Without down-weighting the POS loss ($\lambda_p = 0.2$), the model heavily prioritized POS classification at the expense of lemma generation quality.
Key Results
- Overall Performance: The system did not outperform the baseline UDPipe 1.2 across most metrics. On English EWT, local evaluation showed:
- POS: 90.47% vs. UDPipe's 93.61% (shared task winner: 95.94%).
- Morphological Features (UFeat): 93.95% vs. UDPipe's 94.60% (winner: 96.03%).
- Lemma: 91.51% vs. UDPipe's 95.92% (winner: 97.23%).
- MLAS / BLEX: 62.12 / 66.35 vs. UDPipe's 68.27 / 70.53.
- TIRA Shared Task Bug: An input preprocessing bug on the competition platform duplicated newlines, breaking sentence segmentation and artificially dropping submitted scores (e.g., English EWT BLEX scored 60.73 on TIRA vs. 66.35 locally).
- Non-Indo-European Degradation: The lemmatization architecture failed severely on Arabic PADT (7.42% Lemma score vs. UDPipe’s 87.42%) and Korean GSD (12.87% vs. 87.03%), demonstrating a lack of generalization to root-and-pattern or agglutinative morphologies.
Limitations and Future Work
- Poor Cross-Lingual Generalization: The reverse-positional character decoder relies heavily on suffix-based transformations and fails drastically on non-Indo-European languages.
- Compute and Speed: Adding parallel character-level decoders slowed training down by more than $3\times$.
- Dependency Parsing Not Integrated: True joint parsing was not achieved; dependency parsing was offloaded entirely to UDPipe 1.2.
- Future Directions: The authors highlight hyperparameter optimization, advanced regularization (e.g., recurrent dropout), network redesign for non-concatenative morphology, and extending the model to incorporate full dependency parsing into a single unified network.
Summary updated