An Open-Source Dataset and A Multi-Task Model for Malay Named Entity Recognition

Yingwen Fu, Nankai Lin, Zhihe Yang, Shengyi JiangarXiv 2021 · arXiv

Abstract

Malay is a low-resource language in the field of Natural Language Processing (NLP), particularly lacking in large-scale, high-quality datasets for Named Entity Recognition (NER). This paper introduces MYNER, the first open-source Malay NER dataset, and MTBR, a multi-task model designed to address boundary recognition errors. By leveraging the linguistic similarity between Malay and Indonesian and incorporating a gated bidirectional revision mechanism, the proposed approach achieves a state-of-the-art F1-score of 92.36%.

Introduction

Named Entity Recognition (NER) is a fundamental task in NLP, involving the identification and classification of entities such as people (PER), locations (LOC), and organizations (ORG). While high-resource languages benefit from extensive benchmarks, Malay has historically suffered from data scarcity and significant Boundary Recognition Errors (BRE). BRE occurs when models fail to precisely identify the start and end of an entity span (e.g., misidentifying "University of Malaya" as just "Malaya"). This research addresses these gaps by providing a robust dataset and a novel architectural framework.

Dataset Construction: The MYNER Framework

The creation of the MYNER dataset utilized a "homologous language" strategy, capitalizing on the lexical similarities between Malay and Indonesian. The researchers used existing Indonesian NER datasets as a seed, filtering them through Malay dictionaries and rule-based extractions.

The iterative optimization process for constructing the MYNER dataset using homologous language strategies.

As shown in the framework above, the process involved an iterative optimization loop. A model initially labeled unlabeled Malay sentences, followed by a manual audit of disagreements. The audited data was then used to re-train the model, ensuring high annotation quality across 28,991 sentences (over 384,000 tokens) sourced from major news outlets like Bharian and Malaysiakini.

The MTBR Model Architecture

The Multi-Task Bidirectional Revision (MTBR) model introduces a sophisticated approach to sequence labeling. It utilizes Bahasa-BERT as a contextual encoder and treats Boundary Detection (BD) as an auxiliary task to support the primary NER classification.

Architecture of the MTBR model featuring the multi-task learning framework and bidirectional revision mechanism.

The architecture features two critical components designed to improve precision:

  1. Bi-revision Mechanism: The BD module provides explicit feedback to the NER module. If a boundary is detected with high confidence, the NER module's probability distribution is revised to align with this boundary.
  2. Gated Ignoring Mechanism: To mitigate the risk of error propagation from the auxiliary task, a learned gate controls the influence of the BD module on the final NER prediction, acting as a safety valve against noise.

Experimental Results

The MTBR model was evaluated against several baselines, including standard BERT, mBERT, and ELMO.

  • Performance: MTBR achieved a peak F1-score of 92.36%, outperforming the multi-task BERT baseline by 1.35%.
  • Error Reduction: The model significantly improved boundary precision, reducing the Boundary Recognition Error (BRE) from 3.53% in single-task models to 1.89%.
  • Ablation Study: Removing the revision or gating components resulted in a measurable performance decline, validating the necessity of the multi-task interaction design.

Conclusion and Future Work

This work provides a critical foundation for Malay NLP. By demonstrating that "linguistic cousins" can be used to bootstrap datasets, the authors offer a blueprint for other low-resource languages. Future research will focus on expanding entity categories beyond PER, LOC, and ORG, and testing the model's robustness on informal social media text which often utilizes non-standard slang.

Summary updated

BibTeX
@misc{fu2021open,
  author = {Yingwen Fu and Nankai Lin and Zhihe Yang and Shengyi Jiang},
  title = {{An Open-Source Dataset and A Multi-Task Model for Malay Named Entity Recognition}},
  year = {2021},
  eprint = {2109.01293},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2109.01293},
}
Figure