Rui Cai and Mirella Lapata
Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing (EMNLP-IJCNLP)
Abstract
This paper introduces an end-to-end model for Semantic Role Labeling (SRL) that effectively leverages unlabeled data through a modified version of Cross-View Training (CVT). By jointly training a "sentence learner" for syntactic tasks alongside the primary SRL task, the authors achieve state-of-the-art results on the CoNLL-2009 benchmark and demonstrate significant performance gains across multiple languages.
Introduction
Semantic Role Labeling (SRL) is a fundamental task in natural language processing that involves identifying the semantic relationships between predicates and their arguments—essentially determining "who did what to whom." While neural models have significantly advanced the field, they remain heavily dependent on large-scale, high-quality human annotations. Such data is expensive to produce and often unavailable for low-resource languages or niche domains.
To address this, researchers have explored semi-supervised learning to utilize the vast amounts of available unlabeled text. However, previous "pipeline" approaches often relied on external tools for part-of-speech (POS) tagging or dependency parsing. This dependency creates a bottleneck: errors from external tools propagate to the SRL model, and the tools themselves may not exist for all languages. This paper proposes a self-sufficient, end-to-end architecture that internalizes syntactic learning and utilizes a novel semi-supervised training objective.
Model Architecture
The proposed model consists of two primary components designed to work in tandem:
- The Sentence Learner (Primary Encoder): This component is a bidirectional LSTM (BiLSTM) that performs auxiliary syntactic tasks, including POS tagging, dependency parsing, and predicate identification. By learning these tasks internally, the model generates "multi-task hidden features" that capture structural information more effectively than general-purpose pre-trained embeddings.
- The Semantic Role Labeler (High-level Encoder): This higher-level BiLSTM takes the features from the sentence learner and uses a biaffine scorer to predict the semantic roles between predicates and potential arguments.
Predicate-Aware Cross-View Training
The core innovation of this work is the adaptation of Cross-View Training (CVT) for the SRL task. In standard CVT, a "teacher" model makes predictions on unlabeled data, and several "student" modules are trained to match these predictions using restricted views of the input (e.g., only the left or right context).
The authors found that standard CVT is insufficient for SRL because the relationship between a word and its predicate is the most critical piece of information. To solve this, they introduced Predicate-Aware CVT. In this framework, the student modules are given views that are restricted relative to the predicate's position:
- Backward/Past Views: For words appearing before the predicate, the model uses context that excludes certain preceding information.
- Forward/Future Views: For words appearing after the predicate, the model uses context that excludes certain following information.
Crucially, when training on unlabeled data, the model must identify potential predicates. The authors discovered that randomly selecting candidate predicates from the unlabeled text yielded better results than selecting predicates the model was most confident about. This strategy prevents the model from over-fitting to common verbs and encourages it to learn a broader range of semantic structures.
Experimental Results
The model was evaluated on the CoNLL-2009 benchmark across four languages: English, Chinese, German, and Spanish.
- State-of-the-Art Performance: The model achieved a new record F1 score of 91.2% on the English in-domain test set, outperforming both single-model and ensemble-based predecessors.
- Robustness and Generalization: On the out-of-domain "Brown" test set, CVT improved the F1 score from 81.6% to 82.5%, demonstrating that semi-supervised training significantly enhances the model's ability to generalize to new text styles.
- Multilingual Efficacy: The approach proved highly portable, showing consistent performance gains across all tested languages without requiring language-specific tuning or external parsers.
Conclusion
By internalizing syntactic feature learning and adapting Cross-View Training to be predicate-aware, this research provides a powerful framework for semi-supervised SRL. The model's ability to outperform existing systems while remaining self-sufficient makes it a promising candidate for application in low-resource settings and complex, multi-task NLP pipelines. Future work may explore annotation projection and the use of even weaker supervision, such as training with only dependency labels.
Summary updated