TL;DR
This paper introduces MENmBERT and MENBERT, novel pre-trained language models specifically designed for Malaysian English, a low-resource creole language. By fine-tuning these models on a custom dataset, the researchers demonstrate significant improvements in Named Entity Recognition (NER) and Relation Extraction (RE) tasks, paving the way for better NLP applications for this unique linguistic variant.
What Problem Does This Paper Address?
Large Language Models (LLMs) and Pre-trained Language Models (PLMs) like BERT have revolutionized Natural Language Processing (NLP). However, their effectiveness often diminishes when applied to low-resource languages or language varieties that deviate significantly from standard English. Malaysian English (Manglish) is a prime example: it's a creole language heavily influenced by Malay, Chinese, and Tamil, exhibiting unique morphosyntactic adaptations and code-switching patterns. This linguistic complexity makes it challenging for standard PLMs, even multilingual ones, to accurately process and understand Manglish text, leading to underperformance in downstream NLP tasks like Named Entity Recognition (NER) and Relation Extraction (RE). The paper aims to bridge this gap by developing and evaluating PLMs tailored for Malaysian English.
Methods
The core of this research lies in adapting existing PLM architectures to the specific characteristics of Malaysian English. The authors propose two main strategies:
- Further Pre-training (FP): This involves taking an existing pre-trained model (like
bert-base-casedorbert-base-multilingual-cased) and continuing its training on a corpus of Malaysian English text. This allows the model to learn the nuances of the target language while retaining its general linguistic knowledge. - MENBERT-FP: Further pre-trained
bert-base-casedon the MEN Corpus. - MENmBERT-FP: Further pre-trained
bert-base-multilingual-casedon the MEN Corpus. - Pre-training from Scratch (SC): This involves training a PLM from the ground up using only the Malaysian English corpus.
- MENBERT-SC: Pre-trained
bert-base-casedfrom scratch on the MEN Corpus.
After pre-training, these models, along with baseline models (bert-base-cased and bert-base-multilingual-cased), are fine-tuned on specific NLP tasks: Named Entity Recognition (NER) and Relation Extraction (RE). Hyperparameter optimization was performed using a grid-based search to find the best configurations for fine-tuning.
Datasets
The primary dataset used in this research is the Malaysian English News Article (MEN) Dataset.
- MEN Corpus: Used for pre-training the MENmBERT and MENBERT models. It comprises 14,320 articles.
- MEN-Dataset (for fine-tuning): This dataset contains 200 news articles with 6,061 annotated entities and 4,095 relation instances. For NER, it has 12 entity labels. For RE, it adapts labels from prominent datasets like DocRED (84 labels) and ACE-2005 (16 labels), focusing on DocRED labels due to their prevalence. The dataset is split into training (75%), validation (15%), and testing (10%) sets.
Key Insights and Ideas
- Language-Specific Pre-training is Crucial: The results strongly suggest that pre-training or further pre-training PLMs on language-specific corpora, especially for creole or low-resource languages, significantly boosts performance on downstream tasks.
- MENmBERT-FP Outperforms Multilingual Baselines: The MENmBERT-FP model, which is a further pre-trained multilingual model, shows a notable improvement over the general
bert-base-multilingual-casedmodel, indicating that tailoring even a multilingual model to a specific dialect is beneficial. - Code-Switching and Morphosyntactic Nuances are Key Challenges: The underperformance of standard models highlights the difficulty in capturing the unique linguistic features of Malaysian English, such as loanwords, compound words, and code-switching.
- Further Pre-training is More Effective than Training from Scratch (for this task): MENBERT-FP and MENmBERT-FP generally outperform MENBERT-SC, suggesting that leveraging existing general linguistic knowledge from large pre-trained models and adapting it is more efficient than starting from scratch for this specific dataset and tasks.
Key Results
Named Entity Recognition (NER):
* MENmBERT-FP achieved the highest F1-Score of 0.831, outperforming bert-base-multilingual-cased (0.819) by +1.52%.
* The paper notes a significant improvement for 6 out of 11 entity labels.
* MENBERT-FP achieved an F1-Score of 0.778, showing a +1.94% improvement over bert-base-cased (0.763).
* MENBERT-SC performed poorly, with an F1-Score of 0.316, indicating that training from scratch without a large general corpus was not effective for this task.
Relation Extraction (RE):
* MENmBERT-FP achieved the highest F1-Score of 0.353, showing a significant improvement of +33.21% compared to mBERT-base-cased (0.2375).
* MENBERT-FP achieved an F1-Score of 0.3105, a +19.33% improvement over bert-base-cased (0.2632).
* The authors note that while MENmBERT-FP shows promise for RE, the overall performance is lower than reported results on other document-level RE datasets, possibly due to the smaller size of the MEN-Dataset's annotated instances.
Limitations and Future Work
Limitations: * The authors acknowledge that the overall performance improvement for NER, while present, is not always dramatic. * The performance of fine-tuned RE models on the MEN-Dataset is lower compared to established RE benchmarks, suggesting that the dataset size or complexity might be a limiting factor. * MENBERT-SC, trained from scratch, showed poor performance in both NER and RE tasks, indicating that this approach might not be optimal for low-resource settings without massive computational resources and data.
Future Work: * Explore data augmentation techniques to expand the MEN-Dataset and improve model training, especially for RE. * Investigate other downstream NLP tasks beyond NER and RE to further evaluate the utility of MENmBERT and MENBERT. * Further optimize RE performance within the context of the MEN-Dataset, potentially by exploring different model architectures or training strategies. * The authors plan to make the MENmBERT and MENBERT models and the MEN-Dataset publicly available to facilitate further research in Malaysian English NLP.
Summary updated