A Focused Study to Compare Arabic Pre-training Models on Newswire IE Tasks

Wuwei Lan, Yang Chen, Wei Xu, Alan Ritter

Abstract

Multilingual pre-trained Transformers, such as mBERT and XLM-RoBERTa, have enabled effective cross-lingual zero-shot transfer. However, their performance on Arabic information extraction (IE) tasks is not well studied. We introduce GigaBERT, a customized bilingual BERT designed specifically for Arabic NLP and English-to-Arabic zero-shot transfer. We evaluate GigaBERT on four IE tasks: named entity recognition (NER), part-of-speech (POS) tagging, argument role labeling (ARL), and relation extraction (RE). GigaBERT significantly outperforms mBERT, XLM-RoBERTa, and AraBERT in both supervised and zero-shot settings.

Introduction

Multilingual Transformers have demonstrated impressive zero-shot cross-lingual transfer capabilities, allowing models fine-tuned on English data to be evaluated on Arabic without target-language training examples. However, Arabic IE performance often lags behind supervised models. English and Arabic present a particular challenge due to their entirely different scripts, which limits natural "anchor points" (like shared subwords) between the languages. This study investigates whether a dedicated, domain-specific bilingual model can bridge this gap more effectively than massive multilingual models.

Methodology

GigaBERT utilizes a BERT-base architecture pre-trained from scratch on English and Arabic text. The approach diverges from standard multilingual pre-training in three key ways:

  • Custom Unified Vocabulary: Instead of the skewed vocabularies of mBERT or XLM-R, GigaBERT uses a 50k WordPiece vocabulary specifically balanced for English and Arabic.
  • Code-Switched Data Augmentation: To force the model to learn cross-lingual alignments across different scripts, synthetic code-switched data was generated. Using bilingual dictionaries, up to 30% of words in a sentence were randomly replaced with their translations.
  • Domain Adaptation: Pre-training focused heavily on the newswire domain using Gigaword (v5), Wikipedia, and the Arabic Oscar corpus.

Results and Discussion

GigaBERT-v4 achieves state-of-the-art results in zero-shot transfer from English to Arabic. As shown in the performance comparison below, GigaBERT-v4 significantly outperforms mBERT and XLM-R across most IE tasks.

Table 3: Performance of various models on IE tasks in supervised (en, ar) and zero-shot (en -> ar) settings.

Key Insights

  • Bilingual Advantage: A carefully tuned bilingual model with a balanced vocabulary can outperform massive models trained on 100+ languages for specific, challenging language pairs.
  • Impact of Code-Switching: Injecting synthetic code-switched text during pre-training creates artificial "anchor points" that dramatically improve zero-shot transfer. Comparing GigaBERT-v4 to the non-code-switched v3 shows a significant jump in zero-shot ARL (from 51.0 to 56.1 F1) and RE (from 46.8 to 48.2 F1).
  • Semantic Alignment: Analysis of the embedding space shows that GigaBERT learns stronger language-agnostic representations. As illustrated in the figure below, GigaBERT successfully clusters parallel English-Arabic sentences (bitext) together while keeping random pairs distinct, a task where mBERT and XLM-R show less separation.
Figure 1: Cosine similarity of parallel (bitext) vs. random sentence pairs across layers for XLM-R, GigaBERT, and mBERT.

Conclusion

GigaBERT demonstrates that domain-specific bilingual pre-training and synthetic code-switching are highly effective for cross-lingual transfer in challenging language pairs like English-Arabic. The model not only excels in zero-shot transfer but also outperforms dedicated monolingual models like AraBERT in supervised Arabic tasks.

Summary updated

Abstract

Multilingual pre-trained Transformers, such as mBERT (Devlin et al., 2019) and XLM-RoBERTa (Conneau et al., 2020a), have been shown to enable the effective cross-lingual zero-shot transfer. However, their performance on Arabic information extraction (IE) tasks is not very well studied. In this paper, we pre-train a customized bilingual BERT, dubbed GigaBERT, that is designed specifically for Arabic NLP and English-to-Arabic zero-shot transfer learning. We study GigaBERT's effectiveness on zero-short transfer across four IE tasks: named entity recognition, part-of-speech tagging, argument role labeling, and relation extraction. Our best model significantly outperforms mBERT, XLM-RoBERTa, and AraBERT (Antoun et al., 2020) in both the supervised and zero-shot transfer settings. We have made our pre-trained models publicly available at https://github.com/lanwuwei/GigaBERT.
BibTeX
@inproceedings{lan2020focused,
  author = {Wuwei Lan and Yang Chen and Wei Xu and Alan Ritter},
  title = {{A Focused Study to Compare Arabic Pre-training Models on Newswire IE Tasks}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2020},
  doi = {10.18653/v1/2020.emnlp-main.382},
}
Figure