Building a semantic role labelling system for Vietnamese

Thai-Hoang Pham, Xuan-Khoai Pham, Phuong Le-Hong

TL;DR

Researchers have developed the first Semantic Role Labelling (SRL) system specifically for the Vietnamese language. By moving away from standard English-centric algorithms and introducing a custom constituent extraction method alongside novel linguistic features, the team achieved a 73.53% $F_1$ score, providing a robust open-source baseline for Vietnamese natural language understanding.

What Problem Does This Paper Address?

Semantic Role Labelling (SRL) is the "Who did What to Whom, When, and Why" of natural language processing. It identifies the predicates (usually verbs) in a sentence and classifies their associated arguments (the actors, objects, and circumstances). While SRL is well-developed for languages like English and Chinese, Vietnamese—an isolating language with distinct syntactic structures—lacked a dedicated system.

The authors discovered that simply "porting" English SRL techniques to Vietnamese fails spectacularly. Specifically, the standard "node-mapping" algorithm used to identify potential arguments in a sentence tree only achieved a 35.84% $F_1$ score on Vietnamese data. The gap was clear: Vietnamese NLP needed a system built for its own unique linguistic rules.

Methods

The authors followed the traditional two-step SRL pipeline but overhauled the internal logic for the Vietnamese context:

  1. Argument Identification: Instead of the standard 1-1 node-mapping, they developed a new Constituent Extraction Algorithm. This algorithm traverses bracketed parse trees by starting at the predicate and moving upward to the root, intelligently collecting sibling nodes and their children based on phrase types and function tags.
  2. Argument Classification: Once candidates are identified, the system classifies them into roles (e.g., Arg0 for agent, ArgM-TMP for time). They tested two statistical models: Maximum Entropy (ME) and Support Vector Machines (SVM).
  3. Feature Engineering: Beyond standard features (like phrase type or head words), they introduced:
  4. Function Tags: Labels indicating the grammatical role (e.g., subject, object).
  5. Word Clusters: Using Gaussian mixture analysis on word embeddings to group similar words, which helps the model handle "unseen" words in small datasets.
  6. Predicate Type: Identifying if the predicate is a verb, noun, adjective, or preposition.

Datasets

The study utilized the Vietnamese PropBank.

  • Scale: It contains roughly 5,000 annotated sentences—significantly smaller than the English PropBank (~50,000 sentences).
  • Complexity: Vietnamese PropBank uses 25 semantic roles (compared to English's 21), leading to higher data sparseness.
  • Refinement: The authors performed extensive manual cleaning to fix annotation and encoding errors, resulting in a final gold-standard set of 4,800 simple sentences.

Key Insights and Ideas

  • English Methods Aren't Universal: The failure of the 1-1 node-mapping algorithm (35.84% $F_1$) highlights that syntactic-semantic mappings in isolating languages like Vietnamese are fundamentally different from inflectional languages like English.
  • Function Tags are High-Value: Adding function tags was the single most impactful feature improvement, boosting the $F_1$ score by approximately 4%.
  • Clustering Mitigates Sparsity: In a small dataset (5k sentences), the model often encounters words it hasn't seen during training. Using word clusters (grouping words by semantic similarity) allows the model to generalize better.
  • SVM > ME: For the classification task, SVMs significantly outperformed Maximum Entropy models, particularly when using a 1-step classification strategy.

Key Results

  • Identification Step: The new extraction algorithm achieved an 83.63% $F_1$ score, a massive leap from the 35.84% achieved by the baseline English-style algorithm.
  • Overall System: The final system reached a 73.53% $F_1$ score using the SVM classifier.
  • Baseline Comparison: The baseline system (using standard English techniques) only managed a 6.20% $F_1$ score on the Vietnamese dataset, proving that the authors' custom approach was essential.

Limitations and Future Work

The primary limitation is the dataset size and complexity. The current system was trained and tested on "simple" sentences (those with only one predicate). Complex sentences with multiple nested predicates remain a challenge due to inconsistencies in the original PropBank annotations.

Future Directions: * Enlarging the Corpus: More data is needed to reach the performance levels seen in English SRL. * Joint Modelling: Moving from a two-step pipeline to a joint model that identifies and classifies arguments simultaneously. * Advanced Inference: Exploring techniques like Integer Linear Programming (ILP) to ensure that the predicted semantic roles are globally consistent (e.g., ensuring a sentence doesn't have two "Arg0" agents where only one is allowed).

Summary updated

Abstract

Semantic role labelling (SRL) is a task in natural language processing which detects and classifies the semantic arguments associated with the predicates of a sentence. It is an important step towards understanding the meaning of a natural language. There exists SRL systems for well-studied languages like English, Chinese or Japanese but there is not any such system for the Vietnamese language. In this paper, we present the first SRL system for Vietnamese with encouraging accuracy. We first demonstrate that a simple application of SRL techniques developed for English could not give a good accuracy for Vietnamese. We then introduce a new algorithm for extracting candidate syntactic constituents, which is much more accurate than the common node-mapping algorithm usually used in the identification step. Finally, in the classification step, in addition to the common linguistic features, we propose novel and useful features for use in SRL. Our SRL system achieves an $F_1$ score of 73.53\% on the Vietnamese PropBank corpus. This system, including software and corpus, is available as an open source project and we believe that it is a good baseline for the development of future Vietnamese SRL systems.
BibTeX
@inproceedings{pham2015building,
  author = {Thai-Hoang Pham and Xuan-Khoai Pham and Phuong Le-Hong},
  title = {{Building a semantic role labelling system for Vietnamese}},
  booktitle = {ICDIM},
  year = {2015},
  url = {https://openreview.net/forum?id=6uM-T8P5ue},
}

Loading related content...

Figure