A Bayesian Model of Multilingual Unsupervised Semantic Role Induction

James Henderson, Nikhil Garg, James Henderson 0001arXiv 2016 · arXiv

Understanding "who did what to whom" in a sentence—a task known as Semantic Role Labeling (SRL)—is a cornerstone of natural language understanding. While supervised systems achieve high accuracy, they rely on expensive, human-annotated datasets like PropBank, which are unavailable for most of the world's languages. This paper proposes a novel Bayesian framework for unsupervised Semantic Role Induction (SRI) that leverages parallel corpora to share semantic information across languages.

The Multilingual Bayesian Framework

The core of the proposed approach is a generative Bayesian model that treats SRI as a clustering problem. The model identifies predicates and clusters their arguments into semantic roles based on features such as dependency relations, head words, and POS tags.

To incorporate multilingual information, the authors introduce Cross-lingual Latent Variables (CLVs). When head words of arguments in a parallel sentence (e.g., English and German) are aligned via word-alignment tools, a CLV is instantiated as a common "parent" to the role labels in both languages. This structure encourages the model to assign consistent semantic roles to aligned arguments across the language pair.

The joint Bayesian model for multilingual semantic role induction, showing cross-lingual latent variables (C) connecting Language 1 and Language 2.

Because the model is generative, it provides a flexible framework for various learning scenarios. By "clamping" specific variables during inference (using Collapsed Gibbs Sampling), the researchers can evaluate the model in unsupervised, semi-supervised, or annotation projection settings without altering the underlying architecture.

Experimental Results and Key Insights

The model was evaluated using the CoNLL 2009 gold-standard datasets for English (EN) and German (DE), supplemented by the Europarl parallel corpus (1.5 million sentence pairs). Performance was measured using Purity (PU), Collocation (CO), and F1 scores.

The "Data Volume" Effect

One of the most striking findings was that the performance boost attributed to multilingual learning often stemmed from the sheer volume of additional data rather than the cross-lingual alignments themselves. As shown in the table below, adding Europarl data to a monolingual German model improved the F1 score from 82.24 to 82.70. Transitioning to a full multilingual model with explicit alignments only provided a marginal further increase to 82.79.

Performance comparison across different training datasets and model configurations, highlighting the impact of additional unlabeled data (EP) versus multilingual alignments.

The Efficiency Gap and Alignment Sparsity

The study also highlighted two major challenges for unsupervised SRI:

  1. The 10% Threshold: The authors observed that providing gold labels for just 10% of the training data (approx. 4,000 sentences) allowed a simple supervised baseline to outperform complex unsupervised multilingual models. This suggests that current unsupervised methods still struggle to match the efficiency of even sparse human supervision.
  2. Alignment Sparsity: A significant bottleneck is the sparsity of alignments; only 8% of English and 17% of German arguments were successfully aligned in the parallel text. This limited signal restricts the potential for cross-lingual information transfer.

Conclusion

While the Bayesian model successfully demonstrates that parallel corpora can improve semantic role induction, the results suggest that the primary benefit of such corpora in current frameworks is the provision of more monolingual evidence. Future work aims to address alignment sparsity and explore more sophisticated joint modeling of SRI and word alignment to better exploit the rich semantic signals present in multilingual data.

Summary updated

Abstract

We propose a Bayesian model of unsupervised semantic role induction in multiple languages, and use it to explore the usefulness of parallel corpora for this task. Our joint Bayesian model consists of individual models for each language plus additional latent variables that capture alignments between roles across languages. Because it is a generative Bayesian model, we can do evaluations in a variety of scenarios just by varying the inference procedure, without changing the model, thereby comparing the scenarios directly. We compare using only monolingual data, using a parallel corpus, using a parallel corpus with annotations in the other language, and using small amounts of annotation in the target language. We find that the biggest impact of adding a parallel corpus to training is actually the increase in mono-lingual data, with the alignments to another language resulting in small improvements, even with labeled data for the other language.
BibTeX
@misc{henderson2016bayesian,
  author = {James Henderson and Nikhil Garg and James Henderson},
  title = {{A Bayesian Model of Multilingual Unsupervised Semantic Role Induction}},
  year = {2016},
  eprint = {1603.01514},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1603.01514},
}
Figure