Towards Semi-Supervised Learning for Deep Semantic Role Labeling

Sanket Vaibhav Mehta, Jay Yoon Lee, Jaime G. Carbonell

Abstract

Neural models have shown several state-of-the-art performances on Semantic Role Labeling (SRL). However, the neural models require an immense amount of semantic-role corpora and are thus not well suited for low-resource languages or domains. The paper proposes a semi-supervised semantic role labeling method that outperforms the state-of-the-art in limited SRL training corpora. The method is based on explicitly enforcing syntactic constraints by augmenting the training objective with a syntactic-inconsistency loss component and uses SRL-unlabeled instances to train a joint-objective LSTM. On CoNLL-2012 English section, the proposed semi-supervised training with 1%, 10% SRL-labeled data and varying amounts of SRL-unlabeled data achieves +1.58, +0.78 F1, respectively, over the pre-trained models that were trained on SOTA architecture with ELMo on the same SRL-labeled data. Additionally, by using the syntactic-inconsistency loss on inference time, the proposed model achieves +3.67, +2.1 F1 over pre-trained model on 1%, 10% SRL-labeled data, respectively.

Introduction

Semantic Role Labeling (SRL) is a fundamental task in natural language processing that identifies the predicate-argument structure of a sentence, answering "who did what to whom." While deep learning has pushed the boundaries of SRL performance, these models typically require vast amounts of human-annotated data. This dependency creates a significant barrier for low-resource languages and specialized domains.

Historically, SRL systems relied heavily on syntactic parse trees. However, modern end-to-end neural models often bypass syntax, relying instead on large-scale data to learn structural patterns. This research explores whether re-introducing syntactic constraints can improve model robustness in data-scarce environments and whether unlabeled sentences with available syntactic parses can be leveraged to enhance SRL performance.

Methodology

The authors propose a semi-supervised approach built upon a state-of-the-art Bi-LSTM architecture with ELMo embeddings. The core innovation is the Syntactic Inconsistency Loss (SI-Loss).

Syntactic Inconsistency Loss

In a linguistically sound SRL labeling, argument spans should generally align with constituents in a syntactic parse tree. SI-Loss quantifies the disagreement between the model's predicted spans and the provided syntactic parse. By incorporating this loss into the training objective, the model is penalized for predicting spans that do not correspond to syntactic constituents.

Semi-Supervised Learning (SSL)

Because SI-Loss only requires a syntactic parse and the model's own predictions—rather than gold semantic labels—it can be applied to unlabeled data. The authors utilize a joint objective: standard cross-entropy loss for labeled data and SI-Loss for both labeled and unlabeled data. This allows the model to learn structural consistency from a much larger pool of text than what is available in annotated SRL corpora.

Experimental Setup

The method was evaluated on the CoNLL-2012 dataset. To simulate low-resource conditions, the researchers created training subsets using only 1% and 10% of the available labeled SRL data. They then supplemented these small sets with varying amounts of the remaining data treated as "unlabeled" (but parsed) input.

Results and Discussion

The semi-supervised approach yielded significant improvements, particularly in the most data-constrained scenarios. As shown in the results, increasing the amount of unlabeled data (denoted as J1x, J5x, J10x) consistently improved F1 scores and reduced the rate of syntactic disagreement.

Performance of semi-supervised models (J) compared to baselines (B) across different amounts of unlabeled data.

Key Findings

  • Performance Gains: With only 1% labeled data, the SSL model (B1-J5x) achieved a +1.58 F1 improvement over the baseline. With 10% labeled data, the improvement was +0.78 F1 (B10-J10x).
  • Syntactic Consistency: Models trained with the joint objective produced fewer "illegal" spans, with disagreement rates dropping by up to 8.8% compared to baselines.
  • Inference-Time Constraints: The benefits of enforcing syntax during training are complementary to those gained during inference. By applying syntactic constraints at test time (using gradient-based inference), the total gains over the baseline reached +3.67 F1 for the 1% data scenario and +2.1 F1 for the 10% scenario.
Comparison of decoding methods, showing the additive benefits of gradient-based inference with syntactic constraints.

Conclusion

This work demonstrates that syntactic parse trees can serve as a powerful "teacher" for neural SRL models, especially when labeled data is scarce. By using syntax as a structural constraint rather than just an input feature, models can effectively learn from unlabeled data. Future research may explore the limits of this approach with even larger unlabeled datasets and its applicability to languages where high-quality parsers are less common.

Summary updated

Abstract

Neural models have shown several state-of-the-art performances on Semantic Role Labeling (SRL). However, the neural models require an immense amount of semantic-role corpora and are thus not well suited for low-resource languages or domains. The paper proposes a semi-supervised semantic role labeling method that outperforms the state-of-the-art in limited SRL training corpora. The method is based on explicitly enforcing syntactic constraints by augmenting the training objective with a syntactic-inconsistency loss component and uses SRL-unlabeled instances to train a joint-objective LSTM. On CoNLL-2012 English section, the proposed semi-supervised training with 1%, 10% SRL-labeled data and varying amounts of SRL-unlabeled data achieves +1.58, +0.78 F1, respectively, over the pre-trained models that were trained on SOTA architecture with ELMo on the same SRL-labeled data. Additionally, by using the syntactic-inconsistency loss on inference time, the proposed model achieves +3.67, +2.1 F1 over pre-trained model on 1%, 10% SRL-labeled data, respectively.
BibTeX
@inproceedings{mehta2018towards,
  author = {Sanket Vaibhav Mehta and Jay Yoon Lee and Jaime G. Carbonell},
  title = {{Towards Semi-Supervised Learning for Deep Semantic Role Labeling}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2018},
  url = {https://openreview.net/forum?id=B1ZJKzMOZS},
}
Figure