Semi-Supervised Semantic Role Labeling with Cross-View Training

Rui Cai, Mirella LapataEMNLP 2019 · OpenReview

Rui Cai and Mirella Lapata
Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing (EMNLP-IJCNLP)

Abstract

This paper introduces an end-to-end model for Semantic Role Labeling (SRL) that effectively leverages unlabeled data through a modified version of Cross-View Training (CVT). By jointly training a "sentence learner" for syntactic tasks alongside the primary SRL task, the authors achieve state-of-the-art results on the CoNLL-2009 benchmark and demonstrate significant performance gains across multiple languages.

Introduction

Semantic Role Labeling (SRL) is a fundamental task in natural language processing that involves identifying the semantic relationships between predicates and their arguments—essentially determining "who did what to whom." While neural models have significantly advanced the field, they remain heavily dependent on large-scale, high-quality human annotations. Such data is expensive to produce and often unavailable for low-resource languages or niche domains.

To address this, researchers have explored semi-supervised learning to utilize the vast amounts of available unlabeled text. However, previous "pipeline" approaches often relied on external tools for part-of-speech (POS) tagging or dependency parsing. This dependency creates a bottleneck: errors from external tools propagate to the SRL model, and the tools themselves may not exist for all languages. This paper proposes a self-sufficient, end-to-end architecture that internalizes syntactic learning and utilizes a novel semi-supervised training objective.

Model Architecture

The proposed model consists of two primary components designed to work in tandem:

  1. The Sentence Learner (Primary Encoder): This component is a bidirectional LSTM (BiLSTM) that performs auxiliary syntactic tasks, including POS tagging, dependency parsing, and predicate identification. By learning these tasks internally, the model generates "multi-task hidden features" that capture structural information more effectively than general-purpose pre-trained embeddings.
  2. The Semantic Role Labeler (High-level Encoder): This higher-level BiLSTM takes the features from the sentence learner and uses a biaffine scorer to predict the semantic roles between predicates and potential arguments.
Overview of the end-to-end SRL architecture, featuring the Sentence Learner for auxiliary syntactic tasks and the high-level Semantic Role Labeler.

Predicate-Aware Cross-View Training

The core innovation of this work is the adaptation of Cross-View Training (CVT) for the SRL task. In standard CVT, a "teacher" model makes predictions on unlabeled data, and several "student" modules are trained to match these predictions using restricted views of the input (e.g., only the left or right context).

The authors found that standard CVT is insufficient for SRL because the relationship between a word and its predicate is the most critical piece of information. To solve this, they introduced Predicate-Aware CVT. In this framework, the student modules are given views that are restricted relative to the predicate's position:

  • Backward/Past Views: For words appearing before the predicate, the model uses context that excludes certain preceding information.
  • Forward/Future Views: For words appearing after the predicate, the model uses context that excludes certain following information.
The Predicate-Aware CVT framework, showing how auxiliary modules are trained on restricted views of the input sentence relative to the predicate.

Crucially, when training on unlabeled data, the model must identify potential predicates. The authors discovered that randomly selecting candidate predicates from the unlabeled text yielded better results than selecting predicates the model was most confident about. This strategy prevents the model from over-fitting to common verbs and encourages it to learn a broader range of semantic structures.

Experimental Results

The model was evaluated on the CoNLL-2009 benchmark across four languages: English, Chinese, German, and Spanish.

  • State-of-the-Art Performance: The model achieved a new record F1 score of 91.2% on the English in-domain test set, outperforming both single-model and ensemble-based predecessors.
  • Robustness and Generalization: On the out-of-domain "Brown" test set, CVT improved the F1 score from 81.6% to 82.5%, demonstrating that semi-supervised training significantly enhances the model's ability to generalize to new text styles.
  • Multilingual Efficacy: The approach proved highly portable, showing consistent performance gains across all tested languages without requiring language-specific tuning or external parsers.

Conclusion

By internalizing syntactic feature learning and adapting Cross-View Training to be predicate-aware, this research provides a powerful framework for semi-supervised SRL. The model's ability to outperform existing systems while remaining self-sufficient makes it a promising candidate for application in low-resource settings and complex, multi-task NLP pipelines. Future work may explore annotation projection and the use of even weaker supervision, such as training with only dependency labels.

Summary updated

BibTeX
@inproceedings{cai2019semi,
  author = {Rui Cai and Mirella Lapata},
  title = {{Semi-Supervised Semantic Role Labeling with Cross-View Training}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2019},
  url = {https://openreview.net/forum?id=V-K7U5exjUd},
}
Figure