Multi-source synthetic treebank creation for improved cross-lingual dependency parsing

Francis Tyers, Mariya Sheyanova, Aleksandra Martynova, Pavel Stepachev, Konstantin VinogorodskiyEMNLP 2018

TL;DR

This paper introduces a novel method for creating synthetic dependency parsing treebanks for low-resource languages by leveraging multiple related, high-resource languages. By combining machine translation, annotation projection, and a maximum spanning tree algorithm, they achieve state-of-the-art results for Faroese dependency parsing, demonstrating the power of multi-source synthetic data.

What Problem Does This Paper Address?

Dependency parsing is a crucial task in Natural Language Processing (NLP) that involves analyzing the grammatical structure of sentences by identifying relationships between words. Building high-quality dependency parsers typically requires large, manually annotated datasets called treebanks. However, many languages, like Faroese, are "under-resourced," meaning they lack such extensive annotated corpora, making it difficult to train effective parsers. This paper tackles the challenge of improving dependency parsing for low-resource languages by creating synthetic training data derived from related, well-resourced languages.

Methods

The authors propose a multi-source synthetic treebank creation method that builds upon existing techniques:

  • Machine Translation (MT): Faroese sentences are first translated into several related, high-resource languages (Danish, Norwegian Bokmål, Norwegian Nynorsk, and Swedish). This translation process often involves "pivot translation," where an intermediate language (like Norwegian Bokmål) is used to bridge the gap between Faroese and other target languages.
  • Annotation Projection: Once translated, these sentences are parsed using parsers trained on the respective high-resource languages. The resulting dependency structures (trees) are then "projected" back to the original Faroese sentences using word alignments. This process generates multiple potential dependency trees for each Faroese sentence, each derived from a different source language.
  • Maximum Spanning Tree (MST) Algorithm: Instead of simply picking one projected tree or averaging them, the paper's novel contribution is to merge all projected trees into a weighted graph. Each arc (dependency relation) in this graph is weighted by the number of times it appears across the projected trees from different source languages. The Maximum Spanning Tree algorithm is then used to find the "best" tree within this graph, effectively voting for the most consistent dependency structures across the multiple sources.

Datasets

  • Faroese Wikipedia: Used as the source of raw Faroese text. This text was cleaned and processed to create a corpus of 28,862 sentences.
  • Gold Standard Faroese Treebank: A manually annotated treebank of 1,208 sentences (10,002 tokens) was created for evaluation. This treebank follows the Universal Dependencies (UD) v2.0 guidelines.
  • Source Language Treebanks: Existing UD treebanks for Swedish (UD_Swedish-Talbanken), Danish (UD_Danish-DDT), Norwegian Bokmål (UD_Norwegian-Bokmaal), and Norwegian Nynorsk (UD_Norwegian-Nynorsk) were used for training the parsers that generate the projected annotations.

Key Insights and Ideas

  • Leveraging Multiple Sources is Key: The core idea is that combining annotations from several related languages provides a richer and more robust signal than relying on a single source language. This "multi-source" approach helps overcome the limitations of individual machine translation systems or projection artifacts.
  • Maximum Spanning Tree for Consensus: Using MST to find the best tree in a graph of projected annotations is a clever way to aggregate evidence from multiple sources. It's more sophisticated than simple voting and can identify consistent structures even when individual projections might be noisy.
  • Pivot Translation as a Bridge: The use of pivot translation is essential for languages like Faroese, which may not have direct, high-quality machine translation systems to all desired target languages. This allows access to a wider range of related language resources.

Key Results

The paper reports significant improvements in dependency parsing for Faroese:

  • The multi-source synthetic treebank approach achieved state-of-the-art results for Faroese dependency parsing using an off-the-shelf parser.
  • Specifically, the multi-source model achieved an F-measure of 74.49% for POS tagging, 72.90% for LAS (Labeled Attachment Score), and 64.43% for UAS (Unlabeled Attachment Score).
  • These results substantially outperform both the delexicalised parsing baseline and the single-source synthetic treebanking approach. For instance, the LAS score for the multi-source model (72.90%) is significantly higher than the best single-source Norwegian Nynorsk projection (70.62%).

Limitations and Future Work

  • Projection Quality: The authors acknowledge that the projection process is deterministic and relies on the "1-best" alignment from word aligners. This might miss better alignments and requires heuristics to handle imperfect projections.
  • Weighting Scheme: The current weighting scheme is based on raw voting. Future work could explore incorporating more linguistic information (e.g., POS tag distributions, feature agreement) into the arc weighting to improve results.
  • Increasing Tree Diversity: The paper suggests increasing the number of projected trees by using different parsers or more machine translation systems to generate more diverse annotations for the graph.
  • Generalization to Other Language Groups: The authors propose exploring this method for other language families with related, well-resourced languages, such as Upper Sorbian.

Summary updated

BibTeX
@inproceedings{tyers2018multi,
  author = {Francis Tyers and Mariya Sheyanova and Aleksandra Martynova and Pavel Stepachev and Konstantin Vinogorodskiy},
  title = {{Multi-source synthetic treebank creation for improved cross-lingual dependency parsing}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2018},
}
Figure