Genre as Weak Supervision for Cross-lingual Dependency Parsing

Max Müller-Eberstein, Rob van der Goot, Barbara Plank

Introduction

Dependency parsing—the task of mapping the grammatical structure of sentences—is a cornerstone of natural language processing. While parsers for high-resource languages like English are highly accurate, thousands of the world's languages lack the annotated data required to train such models. The standard solution is "zero-shot" transfer: training a parser on a diverse pool of data from other languages and applying it to a new target. However, this approach often ignores a critical variable: genre.

Syntax varies significantly between formal news reports, fictional narratives, and spoken dialogue. This paper argues that "genre" is a powerful, underutilized signal for cross-lingual transfer. By projecting coarse treebank-level genre labels down to the sentence level, the authors achieve state-of-the-art results for several low-resource languages without any target-language training data.

Methodology: Sentence-Level Data Selection

The core challenge is that genre metadata in datasets like Universal Dependencies (UD) is typically applied to entire treebanks, which are often noisy and contain mixed content. To address this, the authors propose moving to sentence-level selection using three primary strategies:

Overview of the genre-based data selection pipeline.
  1. BOOT (Bootstrapping): A classifier is trained on "pure" single-genre treebanks to predict labels for sentences in mixed-genre collections. High-confidence predictions are iteratively added to the training set, refining the model’s understanding of genre across languages.
  2. GMM (Gaussian Mixture Models): Using multilingual BERT (mBERT) embeddings, the system clusters sentences within treebanks and selects those geometrically closest to a small sample of the target language.
  3. LDA (Latent Dirichlet Allocation): This method clusters sentences based on character n-grams rather than deep embeddings. This surface-level lexical approach is particularly effective for languages not well-represented in mBERT’s pre-training.

These methods were evaluated against a META baseline (using all data from any treebank sharing the target genre label) and a SENT baseline (selecting sentences based on embedding similarity without explicit genre labels).

Key Insights: Genre is Cross-lingual

A central finding of this research is that genre characteristics are remarkably consistent across different languages. As shown in the embedding visualizations, mBERT and the refined BOOT model can distinguish between genres like "news," "social," or "spoken" regardless of the underlying language.

t-SNE visualization of genre clusters in mBERT and BOOT embedding spaces.

Furthermore, the study reveals that "less is more." The META baseline, which utilized the largest volume of data, actually performed the worst. This suggests that the noise inherent in coarse, treebank-level labels is detrimental to model performance. By using genre as a weak supervisor to filter the training pool, the models learned more robust syntactic patterns that transferred better to unseen languages.

Results and Performance

The proposed genre-based selection methods achieved new state-of-the-art (SOTA) results for Cantonese, Chukchi, and Faroese in zero-shot settings. Quantitatively, the GMM and LDA methods achieved an average Labeled Attachment Score (LAS) of ~38.7, significantly outperforming the random selection baseline (36.5) and the treebank-level META baseline (34.1).

Beyond accuracy, these methods offer massive gains in efficiency. By selecting a targeted subset of approximately 80,000 instances, parsers could be trained in roughly 90 minutes. In contrast, training on the full META dataset required 55 hours and yielded inferior results.

Conclusion and Future Directions

This work demonstrates that genre-aware data selection is a highly effective and efficient strategy for low-resource dependency parsing. While the current study focused on single-genre targets, future research will explore multi-genre mixtures and more data-driven, unsupervised discovery of "universal registers" to further refine cross-lingual transfer.

Summary updated

BibTeX
@inproceedings{mullereberstein2021genre,
  author = {Max Müller-Eberstein and Rob van der Goot and Barbara Plank},
  title = {{Genre as Weak Supervision for Cross-lingual Dependency Parsing}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2021},
  doi = {10.18653/v1/2021.emnlp-main.393},
}
Figure