BERTie Bott's Every Flavor Labels: A Tasty Introduction to Semantic Role Labeling for Galician

Micaella Bruton, Meriem BeloucifEMNLP 2023 · DOI

Abstract

Semantic Role Labeling (SRL) is a critical task for natural language understanding, yet it remains inaccessible for the vast majority of the world's languages. This paper addresses this gap for Galician, a low-resource Western Ibero-Romance language, by introducing its first-ever SRL dataset. The authors propose a "Verbal Indexing" preprocessing technique to handle complex sentences with multiple predicates. Experimental results demonstrate that leveraging cross-lingual transfer learning with XLM-R and the indexing method yields a strong baseline F1 score of 0.74 for Galician and establishes a new state-of-the-art for Spanish SRL.

Introduction

Semantic Role Labeling (SRL) identifies "who did what to whom" by determining predicates (usually verbs) and their associated arguments (participants like agents or recipients). While essential for downstream tasks, SRL is notoriously data-intensive; currently, only about 0.3% of the world's languages have access to extensive SRL resources like PropBank. Galician, spoken by approximately 4 million people, has historically lacked the necessary annotated corpora to develop robust SRL systems. This work provides the first Galician SRL resource and introduces architectural improvements to better manage sentence complexity.

Methodology

The researchers developed a semi-automatic pipeline to create the Galician SRL dataset, mapping Universal Dependency treebanks to semantic roles via Galician WordNet and English PropBank role sets. The resulting dataset contains nearly 4,000 training sentences and supports up to 13 verbal roots per sentence.

A key technical innovation of the paper is "Verbal Indexing." In sentences with multiple verbs, standard models often struggle to associate arguments with the correct predicate. By assigning unique indices (e.g., r0, r1) to each verb and prefixing argument labels accordingly (e.g., r0:arg0 vs r1:arg0), the model can more effectively disambiguate participants in complex linguistic structures.

Example of the Verbal Indexing method showing how arguments are linked to specific predicates (e.g., r0:arg0).

Experimental Results

The study compared mBERT and XLM-R architectures across various transfer learning scenarios, utilizing high-resource relatives like Spanish, Portuguese, and English. XLM-R consistently outperformed mBERT, a result the authors attribute to the SentencePiece tokenizer. Unlike mBERT’s WordPiece, SentencePiece better preserves Galician morphological units (e.g., correctly handling sexualidade as a single unit rather than splitting it into sexual and idade).

As shown in the results table below, the best-performing model, gal_ensp_XLM-R (pre-trained on English and Spanish), achieved an F1 score of 0.74. Furthermore, applying the Verbal Indexing method to the Spanish CoNLL 2009 dataset resulted in an F1 of 0.83, surpassing the established state-of-the-art baseline by 0.025.

Performance comparison of mBERT and XLM-R models across different language transfer settings.

Key Insights and Complexity Handling

The "Verbal Indexing" method proved to be the "secret sauce" for handling "deep" sentences. While standard models' accuracy typically drops to zero in sentences containing more than six verbs, the indexed models maintained the ability to track arguments accurately up to the ninth verb. Additionally, pre-training on Spanish proved more effective for Galician than Portuguese, likely due to the compatibility of the feature space created by the shared indexing methodology.

Conclusion

This work establishes a strong foundation for Galician SRL, proving that even low-resource languages can achieve high-performance results through innovative preprocessing and strategic transfer learning. Future work will focus on manual validation by native speakers and exploring whether the gains observed in Spanish and Galician are universal across the Ibero-Romance family.

Summary updated

BibTeX
@inproceedings{bruton2023bertie,
  author = {Micaella Bruton and Meriem Beloucif},
  title = {{BERTie Bott's Every Flavor Labels: A Tasty Introduction to Semantic Role Labeling for Galician}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2023},
  doi = {10.18653/v1/2023.emnlp-main.671},
}
Figure