PhoNLP: A joint multi-task learning model for Vietnamese part-of-speech tagging, named entity recognition and dependency parsing

Linh The Nguyen, Dat Quoc NguyenNAACL 2021 · DOI

TL;DR

This paper introduces PhoNLP, the first joint multi-task learning model for Vietnamese Part-of-Speech (POS) tagging, Named Entity Recognition (NER), and dependency parsing. By jointly training these tasks, PhoNLP achieves state-of-the-art results on Vietnamese benchmark datasets, outperforming single-task models and offering a more efficient, unified toolkit for NLP research.

What Problem Does This Paper Address?

Vietnamese Natural Language Processing (NLP) research has seen significant growth, with fundamental tasks like POS tagging, NER, and dependency parsing playing crucial roles in downstream applications. While previous toolkits like VnCoreNLP exist, they are based on traditional machine learning models and are no longer state-of-the-art. The advent of pre-trained language models like PhoBERT has revolutionized NLP, but fine-tuning separate PhoBERT models for each task can lead to large storage requirements. Furthermore, these tasks are inherently related (e.g., POS tags are features for dependency parsing), suggesting that a joint learning approach could be more effective and efficient. This paper addresses the need for a unified, state-of-the-art model that can perform these three core Vietnamese NLP tasks simultaneously, while also being mindful of computational resources.

Methods

PhoNLP employs a multi-task learning architecture built upon a pre-trained BERT-based language model, specifically PhoBERT. The core idea is to leverage a single encoder to generate contextualized word embeddings for the input sentence. These embeddings are then fed into three separate decoding layers, one for each task:

  • POS Tagging: A linear prediction layer followed by a softmax function predicts POS tags for each word. A cross-entropy loss is used for training.
  • NER: The contextualized word embeddings are combined with "soft" POS tag embeddings (derived from the POS tagging layer's probability distribution) and fed into a feed-forward network (FFNN). A linear-chain Conditional Random Field (CRF) predictor then identifies NER labels.
  • Dependency Parsing: Similar to NER, contextualized word embeddings are combined with another set of "soft" POS tag embeddings. This is then processed by FFNNs to generate representations for arc heads and labels, which are fed into a Biaffine classifier to predict dependency arcs and their labels.

The overall training objective is a weighted sum of the individual task losses (POS tagging, NER, and dependency parsing).

Datasets

The experiments were conducted using the following Vietnamese benchmark datasets:

  • VLSP 2013 POS tagging dataset: This dataset was used for POS tagging.
  • VLSP 2016 NER dataset: Used for Named Entity Recognition.
  • VnDT dependency treebank v1.1: Used for dependency parsing.

A significant contribution of this paper is the identification and handling of a data leakage issue in these datasets. Specifically, sentences from the NER and dependency parsing datasets were found to be included in the POS tagging training set, leading to unrealistic evaluation scenarios. The authors address this by re-splitting the VLSP 2013 POS tagging dataset to ensure that validation/test sentences for NER and dependency parsing do not overlap with the POS tagging training set. They also address sentence duplication within the POS tagging dataset.

Key Insights and Ideas

  • Joint Learning Synergy: The paper demonstrates that jointly learning POS tagging, NER, and dependency parsing is not only feasible but also leads to improved performance compared to single-task fine-tuning, likely due to the inherent linguistic relationships between these tasks.
  • "Soft" POS Tag Embeddings: The novel use of "soft" POS tag embeddings, derived from the probability distribution of predicted POS tags, as input features for both NER and dependency parsing layers significantly enhances performance. This allows the model to leverage uncertainty in POS predictions.
  • Data Leakage is a Real Problem: The identification and meticulous handling of data leakage in established Vietnamese NLP benchmarks highlight a critical issue that can inflate reported performance metrics and lead to misleading conclusions. The re-split datasets provide a more realistic evaluation.
  • Efficiency and Resource Savings: By consolidating three tasks into a single model, PhoNLP offers a more compact solution compared to maintaining three separate fine-tuned BERT models, which is crucial for practical applications with limited storage.

Key Results

PhoNLP achieves state-of-the-art results on the Vietnamese benchmark datasets for all three tasks. When compared to a baseline single-task training approach using PhoBERT, PhoNLP consistently outperforms it across POS tagging accuracy, NER F1-score, and dependency parsing LAS/UAS scores. For instance, on the re-split datasets, PhoNLP achieves 96.76% POS tagging accuracy, 94.41% NER F1-score, and 79.11% LAS (dependency parsing accuracy), surpassing the single-task baseline. The paper also provides speed test results, showing PhoNLP can process 15 sentences per second on a CPU and 129 sentences per second on a GPU.

Limitations and Future Work

The authors acknowledge that their current work focuses on Vietnamese. A key direction for future work is to extend PhoNLP to other languages, noting that the toolkit's design is amenable to this by simply swapping the pre-trained language model and providing gold-annotated corpora for the target language. The paper also mentions that the current implementation uses a single PhoBERTbase model, and exploring larger or more advanced pre-trained models could further improve performance.

Summary updated

BibTeX
@inproceedings{nguyen2021phonlp,
  author = {Linh The Nguyen and Dat Quoc Nguyen},
  title = {{PhoNLP: A joint multi-task learning model for Vietnamese part-of-speech tagging, named entity recognition and dependency parsing}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2021},
  doi = {10.18653/v1/2021.naacl-demos.1},
}
Figure