Semi-supervised Semantic Role Labeling Using the Latent Words Language Model

Koen Deschacht, Marie-Francine MoensEMNLP 2009 · OpenReview

Abstract

Semantic Role Labeling (SRL) has proved to be a valuable tool for performing automatic analysis of natural language texts. Currently however, most systems rely on a large training set, which is manually annotated, an effort that needs to be repeated whenever different languages or a different set of semantic roles is used in a certain application. A possible solution for this problem is semi-supervised learning, where a small set of training examples is automatically expanded using unlabeled texts. We present the Latent Words Language Model, which is a language model that learns word similarities from unlabeled texts. We use these similarities for different semi-supervised SRL methods as additional features or to automatically expand a small training set. We evaluate the methods on the PropBank dataset and find that for small training sizes our best performing system achieves an error reduction of 33.27% F1-measure compared to a state-of-the-art supervised baseline.

Introduction

Semantic Role Labeling (SRL) is a critical task in natural language processing that involves identifying the semantic relationships between a predicate and its arguments—essentially answering "who did what to whom." While modern SRL systems are highly capable, they are bottlenecked by their reliance on massive, manually annotated datasets like PropBank. Creating these datasets is prohibitively expensive and time-consuming, especially when expanding to new languages or specialized domains.

To address this "annotation bottleneck," this research explores semi-supervised learning. The goal is to leverage small amounts of labeled data in conjunction with vast quantities of unlabeled text to build robust models that generalize better than purely supervised ones.

Methodology: The Latent Words Language Model

The core of the proposed approach is the Latent Words Language Model (LWLM). Unlike traditional language models that focus on predicting the next word in a sequence, the LWLM introduces a latent (hidden) variable for every word position in a text.

Graphical representation of the Latent Words Language Model (LWLM) architecture.

As shown in Figure 1, the model captures the "meaning" or "type" of a word based on its local context (typically two words to the left and two to the right). By training on a 5-million-word Reuters news corpus, the model learns semantic similarities. For instance, in the context "The ___ broke," the model learns that words like window, glass, and vase share similar latent distributions because they are semantically appropriate for that slot.

Due to the large vocabulary size (over 30,000 words), standard training algorithms like Baum-Welch were computationally infeasible. Instead, the authors employed Gibbs sampling, an approximate inference technique, to make the model training tractable.

Integration into SRL

The researchers evaluated two primary methods for integrating LWLM knowledge into an SRL system (a modified Maximum Entropy Markov Model):

  1. LWFeatures: The probability distributions of the latent words are used directly as additional features in the discriminative classifier.
  2. Automatic Expansion: The LWLM identifies similar sentences in unlabeled data, which are then automatically labeled and added to the training set.

Results and Discussion

The evaluation was conducted using the PropBank dataset (CoNLL 2008 shared task). The results demonstrate that the LWFeatures approach is significantly more effective than automatic expansion or other clustering techniques.

F1-measure comparison across different training set sizes and semi-supervised methods.

As detailed in Figure 2, the performance gains are most pronounced when training data is scarce. With only 5% of the PropBank training data, the supervised baseline achieves a 40.49% F1-measure. The LWFeatures model improves this to 60.29%, representing a 33.27% error reduction. Even with the full 100% training set, the LWFeatures model provides a notable boost from 78.65% to 80.98%.

Key Insights

  • Feature Enrichment vs. Data Expansion: Enriching existing features with "latent" word distributions proved more effective than adding more (potentially noisy) automatically labeled sentences.
  • Robustness: The LWLM approach consistently outperformed Brown Clustering, particularly as the size of the labeled training set increased.
  • Simplicity: The direct use of latent distributions as features provided a clean, powerful signal that allowed the classifier to generalize to words not seen during supervised training.

Conclusion and Future Work

The Latent Words Language Model offers a powerful way to mitigate the data scarcity problem in SRL. By learning semantic similarities from unlabeled text, it allows models to perform significantly better with limited human annotation.

Future research directions include expanding the LWLM context window to capture more distant syntactic dependencies, developing smarter thresholds for automatic expansion, and testing the LWLM features on other NLP tasks like Named Entity Recognition (NER).

Summary updated

Abstract

Semantic Role Labeling (SRL) has proved to be a valuable tool for performing automatic analysis of natural language texts. Currently however, most systems rely on a large training set, which is manually annotated, an effort that needs to be repeated whenever different languages or a different set of semantic roles is used in a certain application. A possible solution for this problem is semi-supervised learning, where a small set of training examples is automatically expanded using unlabeled texts. We present the Latent Words Language Model, which is a language model that learns word similarities from unlabeled texts. We use these similarities for different semi-supervised SRL methods as additional features or to automatically expand a small training set. We evaluate the methods on the PropBank dataset and find that for small training sizes our best performing system achieves an error reduction of 33.27% F1-measure compared to a state-of-the-art supervised baseline.
BibTeX
@inproceedings{deschacht2009semi,
  author = {Koen Deschacht and Marie-Francine Moens},
  title = {{Semi-supervised Semantic Role Labeling Using the Latent Words Language Model}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2009},
  url = {https://openreview.net/forum?id=Sy-RqGf_ZB},
}
Figure