Towards Semi-Supervised Learning for Deep Semantic Role Labeling

Sanket Vaibhav Mehta, Jay Yoon Lee, Jaime G. Carbonell

Abstract

Neural models have shown several state-of-the-art performances on Semantic Role Labeling (SRL). However, these models require an immense amount of semantic-role corpora and are thus not well suited for low-resource languages or domains. This paper proposes a semi-supervised SRL method that outperforms the state-of-the-art in limited SRL training corpora. The method is based on explicitly enforcing syntactic constraints by augmenting the training objective with a syntactic-inconsistency loss component and uses SRL-unlabeled instances to train a joint-objective LSTM. On the CoNLL-2012 English section, the proposed semi-supervised training with 1% and 10% SRL-labeled data achieves +1.58 and +0.78 F1 improvements, respectively, over strong ELMo-based baselines. Additionally, by using the syntactic-inconsistency loss at inference time, the proposed model achieves +3.67 and +2.1 F1 over the pre-trained models on 1% and 10% SRL-labeled data.

Introduction

Semantic Role Labeling (SRL) is a fundamental task in natural language processing that involves identifying the predicate-argument structure of a sentence—essentially answering "who did what to whom." While deep learning has significantly advanced SRL performance, these models are notoriously data-hungry, requiring vast amounts of human-annotated corpora. This dependency creates a bottleneck for low-resource languages and specialized domains where annotated data is scarce.

However, while semantic labels are rare, syntactic information (like parse trees) is often more accessible. This research explores how to leverage this "easier" syntactic information to improve SRL models when labeled semantic data is limited.

Methodology

The core innovation of this work is the Syntactic Inconsistency Loss (SI-Loss). In a linguistically sound sentence, the spans of a semantic argument should align with constituents in a syntactic parse tree. The authors define a "disagreement rate" between the model’s predicted spans and available syntactic parse spans.

Joint Objective and Semi-Supervised Learning

During training, the model minimizes a joint objective:

  1. Standard Cross-Entropy Loss: Measures the accuracy of semantic role labels on annotated data.
  2. SI-Loss: Penalizes the model when it proposes argument spans that do not exist in the syntactic tree.

Crucially, SI-Loss does not require semantic labels. This allows the use of large amounts of unlabeled text. By running this text through a syntactic parser, the resulting trees provide a training signal that teaches the model the "shape" of valid arguments, even in the absence of gold SRL labels.

Key Results

Semi-Supervised Performance

The proposed method shows dramatic improvements in highly constrained settings. By adding unlabeled data with the joint objective, the model significantly outperforms the baselines trained only on limited labeled data.

Performance gains using varying amounts of unlabeled data in 1% and 10% labeled settings.

As shown in the table above, in the 1% labeled data scenario, the model (B1-J5x) achieves a +1.58 F1 improvement over the baseline. Similarly, in the 10% setting, the model (B10-J10x) sees a +0.78 F1 boost. The "average disagreement rate" also drops significantly, indicating that the model learns to respect linguistic boundaries more effectively.

Inference-Time Constraints

Beyond training, the authors explored applying syntactic constraints during the decoding (inference) phase. Traditionally, syntactic constraints are applied as a post-processing step. This work demonstrates that combining a model trained with SI-Loss with gradient-based constrained decoding yields the best results.

Test F1 improvements using syntactic constraints during inference.

When using syntactic constraints at inference time, the model achieves a massive +3.67 F1 boost over the baseline in the 1% data setting (reaching 70.95 F1) and a +2.1 F1 boost in the 10% setting. Notably, the method remains robust even when the syntactic constraints are "noisy" (generated by a parser rather than gold-standard), whereas traditional A* decoding often struggles with such noise.

Conclusion

This research demonstrates that syntax can serve as a powerful "teacher" for SRL models in low-resource environments. By enforcing syntactic consistency during both training and inference, models can generalize better from limited data. Future work may involve extending this approach to dependency-based SRL and scaling to even larger unlabeled datasets to further reduce the reliance on expensive human annotations.

Summary updated

Abstract

Neural models have shown several state-of-the-art performances on Semantic Role Labeling (SRL). However, the neural models require an immense amount of semantic-role corpora and are thus not well suited for low-resource languages or domains. The paper proposes a semi-supervised semantic role labeling method that outperforms the state-of-the-art in limited SRL training corpora. The method is based on explicitly enforcing syntactic constraints by augmenting the training objective with a syntactic-inconsistency loss component and uses SRL-unlabeled instances to train a joint-objective LSTM. On CoNLL-2012 English section, the proposed semi-supervised training with 1%, 10% SRL-labeled data and varying amounts of SRL-unlabeled data achieves +1.58, +0.78 F1, respectively, over the pre-trained models that were trained on SOTA architecture with ELMo on the same SRL-labeled data. Additionally, by using the syntactic-inconsistency loss on inference time, the proposed model achieves +3.67, +2.1 F1 over pre-trained model on 1%, 10% SRL-labeled data, respectively.
BibTeX
@inproceedings{mehta2018towards,
  author = {Sanket Vaibhav Mehta and Jay Yoon Lee and Jaime G. Carbonell},
  title = {{Towards Semi-Supervised Learning for Deep Semantic Role Labeling}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2018},
  url = {https://openreview.net/forum?id=B1ZJKzMOZS},
}
Figure