Abstract
This paper describes the system for our participation in the CoNLL 2017 Shared Task: Multilingual Parsing from Raw Text to Universal Dependencies. In this work, we design a system based on UDPipe1 for universal dependency parsing, where multilingual transition-based models are trained for different treebanks. Our system directly takes raw texts as input, performing several intermediate steps like tokenizing and tagging, and finally generates the corresponding dependency trees. For the special surprise languages for this task, we adopt a delexicalized strategy and predict basing on transfer learning from other related languages. In the final evaluation of the shared task, our system achieves a result of 66.53% in macro-averaged LAS F1-score.
Introduction
The CoNLL 2017 Shared Task presented a significant challenge in natural language processing: parsing raw text into Universal Dependency (UD) trees across a diverse set of languages. Unlike traditional parsing tasks that often provide pre-tokenized and tagged data, this task required systems to handle raw strings. Furthermore, the inclusion of "surprise languages"—languages with no provided training data—necessitated innovative cross-lingual transfer strategies. Our team, Wanghao-ftd-SJTU, developed a robust transition-based pipeline leveraging the UDPipe framework to address these hurdles.
System Architecture
Our system is built upon the modular UDPipe framework, which processes text through three primary stages: tokenization, tagging, and parsing.
- Tokenizer: This component segments raw text into sentences and words. It utilizes simple rules and space-based features to handle the initial processing of raw input.
- Tagger: Using an Averaged Perceptron model, the tagger assigns Universal Part-of-Speech (UPOS) tags and morphological features to the tokenized text.
- Parser: The core of the system is a transition-based parser employing the arc-standard algorithm. It models parsing as a sequence of "shift" and "reduce" operations, predicted by a neural network classifier.
For Surprise Languages, we implemented a delexicalized transfer learning strategy. Because no training data was available for these languages, we stripped lexical information and relied solely on UPOS tags. By mapping a surprise language to a syntactically similar "source" language (e.g., using a Turkish model for Buryat), we were able to predict dependency structures without language-specific lexical knowledge.
Experimental Setup
The system was trained and evaluated using the Universal Dependencies (UD) Release 2.0, which contains over 70 treebanks. The evaluation included "Ordinary" resource sets, parallel test sets across different domains, and four specific surprise languages: Buryat, Kurmanji, North Sami, and Upper Sorbian.
Results and Discussion
In the official CoNLL 2017 evaluation, our system achieved a macro-averaged Labeled Attachment Score (LAS) of 66.53%. The performance varied significantly across different categories of data. For large treebanks with ample training data, the system reached 71.20% LAS. For surprise languages, the delexicalized strategy yielded an average LAS of 34.49%. While lower than the scores for known languages, this represents a successful baseline for zero-shot cross-lingual transfer.
The system demonstrated high accuracy in the early stages of the pipeline, with 98.81% accuracy in tokenization and 90.63% in UPOS tagging. These strong front-end results provided a reliable foundation for the subsequent dependency parsing.
Conclusion
Our transition-based approach proved to be an efficient and effective solution for multilingual parsing from raw text. The use of delexicalized transfer learning highlighted the utility of Universal POS tags as a bridge between related languages. Future work will focus on integrating more advanced neural architectures like LSTMs or Transformers and incorporating external resources such as pre-trained word embeddings to further improve parsing accuracy, especially for low-resource and surprise languages.
Summary updated