TL;DR
The TurkuNLP team presents an end-to-end neural parsing pipeline for the CoNLL 2018 Shared Task, achieving top ranks by integrating established components with a novel neural lemmatization approach. Their system excels in lemmatization, securing first place, and also performs strongly in morphological tagging and syntactic parsing, demonstrating the power of a unified, adaptable pipeline.
What Problem Does This Paper Address?
The CoNLL Universal Dependencies (UD) Shared Tasks aim to evaluate end-to-end parsing systems that can process raw text and output a structured representation of syntactic dependencies. While previous tasks focused primarily on syntactic trees, the 2018 task introduced crucial new metrics for morphological tagging and lemmatization. This expansion highlights the need for robust systems that can handle the full spectrum of linguistic analysis, especially for morphologically rich languages where accurately predicting word forms and their base forms is critical for downstream applications. The challenge lies in building a cohesive pipeline that integrates these diverse tasks effectively and efficiently, particularly under tight development schedules.
Methods
The TurkuNLP pipeline is an end-to-end system designed for modularity and reusability. For segmentation, tagging, and parsing, they largely leverage well-established, high-performing components. Specifically, they use the UDPipe baseline for tokenization and sentence splitting (with custom adjustments for Thai, Breton, and Faroese) and the Dozat et al. (2017) parser, which was the winner of the 2017 shared task. A clever trick is employed for morphological feature tagging: they concatenate the XPOS field with the FEATS field, training the tagger to predict this combined string, and then split it back into the correct columns.
The most novel contribution is their lemmatization component. They frame lemmatization as a sequence-to-sequence translation problem, treating the input word (as a sequence of characters) along with its POS and morphological tags as the source, and the lemma (as a sequence of characters) as the target. This is implemented using the OpenNMT neural machine translation toolkit, featuring a deep attentional encoder-decoder network with bidirectional LSTMs. This approach allows them to leverage powerful neural translation models for a task that has historically been challenging for morphologically complex languages.
Datasets
The primary datasets used are the Universal Dependencies (UD) treebanks, comprising 82 treebanks across 57 languages for the 2018 shared task. Participants were also permitted to use additional resources, including Wikipedia dumps, raw web crawl data, word embeddings, morphological transducers (like Apertium and Giellatekno), and the OPUS parallel corpus. The paper also mentions the use of pre-trained embeddings from the 2017 shared task and Facebook's fastText embeddings. For low-resource languages like Thai, they created synthetic training data by machine translating English treebanks.
Key Insights and Ideas
- Lemmatization as Translation: Framing lemmatization as a sequence-to-sequence task with neural machine translation models proved highly effective, leading to their top performance in this area. This approach is particularly beneficial for morphologically rich languages.
- The Power of Concatenation: The simple yet effective trick of concatenating XPOS and FEATS fields for morphological tagging allowed their parser to leverage richer morphological information, leading to improved LAS scores.
- Modularity for Efficiency: Building an end-to-end pipeline with modular components allowed them to quickly integrate existing state-of-the-art tools while focusing development effort on the novel lemmatization component, crucial given the tight schedule.
- Handling Low-Resource Languages: The paper demonstrates flexible strategies for low-resource languages, including synthetic data generation (Thai) and leveraging morphological transducers (Breton, Faroese), showcasing adaptability.
Key Results
The TurkuNLP system achieved an impressive aggregate ranking, securing 1st place on the metric involving lemmatization (BLEX) and 2nd place on both morphological tagging (MLAS) and syntactic parsing (LAS) among 26 participating teams. Their lemmatizer achieved an accuracy of 77.6% for Breton, significantly outperforming the second-best team. For the overall CoNLL 2018 Shared Task, their system ranked 1st overall in aggregate performance.
Limitations and Future Work
The authors acknowledge that their approach to morphological tagging by concatenating XPOS and FEATS, while effective, is a "simple trick." They also note that for some languages, the XPOS field already contains rich morphological information, and their system did not significantly benefit from adding more in those cases.
Future work could involve exploring the use of the UPOS field for morphological features, as it's a more standardized location. They also suggest that their novel lemmatizer, while effective, could be further improved by incorporating sentence context, which was not done in this implementation. The paper also highlights the ongoing challenge of achieving high performance on very low-resource languages, where even their best efforts for Thai and Breton fell short of typical supervised results.
Summary updated