A Flexible and Easy-to-use Semantic Role Labeling Framework for Different Languages

Quynh Ngoc Thi Do, Artuur Leeuwenberg, Geert Heyman, Marie-Francine MoensCOLING 2018 · OpenReview

Abstract

Semantic Role Labeling (SRL) is a critical task in Natural Language Processing, yet most state-of-the-art models are hyper-optimized for English and require significant manual effort to adapt to other languages. This paper introduces DAMESRL, a modular, open-source framework designed to make deep SRL accessible and customizable across diverse linguistic structures. By enabling a "plug-and-play" approach to neural components, DAMESRL allows researchers to tailor architectures to specific language traits and data constraints. Our evaluation across English, German, and Arabic demonstrates that while self-attention excels in high-resource settings, recurrent models and character-level embeddings are vital for low-resource and morphologically rich languages.

Introduction

Semantic Role Labeling (SRL) aims to identify the predicate-argument structure of a sentence, answering "who did what to whom, where, and when." While deep learning has reduced the need for manual feature engineering, the "one-size-fits-all" approach often fails across different languages. English relies heavily on word order, whereas languages like German and Arabic utilize complex inflections to signal grammatical roles. Furthermore, many languages lack the massive annotated datasets available for English. DAMESRL addresses these challenges by providing a flexible framework to experiment with various neural architectures.

The DAMESRL Framework

The authors propose a four-phase modular pipeline for constructing SRL models, allowing for high degrees of customization.

The DAMESRL modular pipeline, illustrating the four phases of model construction from input to output.
  1. Phase I: Word Representation: Users can combine standard word embeddings with predicate indicators and character-level Bi-LSTM representations. Character-level features are particularly effective for capturing the internal structure (prefixes/suffixes) of words in inflected languages.
  2. Phase II: Sequence Representation: This "engine" layer supports standard Bi-LSTMs, Highway Bi-LSTMs for deeper gradient flow, or modern Self-Attention (Transformer-style) networks.
  3. Phase III: Output Modeling: Hidden states are converted into probabilities using either Softmax or Conditional Random Fields (CRF), the latter being useful for capturing dependencies between adjacent labels.
  4. Phase IV: Inference: The framework offers a choice between fast Greedy decoding and globally optimal Viterbi decoding.

Beyond model construction, DAMESRL includes visualization tools that generate intuitive HTML "grids" to represent semantic roles.

Example of the readable HTML output generated by the framework, visualizing semantic roles for a complex sentence.

Experimental Setup and Datasets

The framework was evaluated on three datasets representing different linguistic families and resource levels:

  • English (CoNLL’05): A large dataset characterized by strict word order.
  • German (CoNLL’09): A medium-sized dataset representing a morphologically rich language.
  • Arabic (CoNLL’12): A small, highly inflected dataset representing a low-resource scenario.

Key Insights and Results

The researchers compared five configurations: HLstm, Char, CRFm, Att, and CharAtt. The CharAtt model (Character embeddings + Self-Attention) emerged as the top performer, achieving F1 scores of 84.9 (English), 73.5 (German), and 55.2 (Arabic).

Several critical insights were gained:

  • Linguistic Nuance: Character-level embeddings significantly improved performance for German and Arabic by providing sub-word cues essential for understanding inflections.
  • Data Sensitivity: While self-attention models were approximately 5x faster to train and highly effective on large datasets, they performed poorly on small datasets (e.g., 2,000 sentences).
  • Recurrent Bias: In low-resource scenarios, LSTMs and CRFs outperformed attention-based models. The inherent sequential bias of recurrent models helps them learn sentence structure more effectively when training data is scarce.

Conclusion

DAMESRL provides a robust platform for end-to-end deep SRL, offering competitive results on English and some of the first deep learning benchmarks for German and Arabic without manual linguistic features. Future work will explore integrating pre-trained language models like BERT to improve performance in low-resource settings and further mitigate the data-hunger of attention-based modules.

Summary updated

BibTeX
@inproceedings{do2018flexible,
  author = {Quynh Ngoc Thi Do and Artuur Leeuwenberg and Geert Heyman and Marie-Francine Moens},
  title = {{A Flexible and Easy-to-use Semantic Role Labeling Framework for Different Languages}},
  booktitle = {International Conference on Computational Linguistics},
  year = {2018},
  url = {https://openreview.net/forum?id=N8cSKSBTEfEU},
}
Figure