An Empirical Study of Pre-trained Transformers for Arabic Information Extraction

Wuwei Lan, Chen Yang, Wei Xu, Alan Ritter

Abstract

Multilingual pre-trained Transformers, such as mBERT and XLM-RoBERTa, have enabled effective cross-lingual zero-shot transfer. However, their performance on Arabic information extraction (IE) tasks is not well studied. In this paper, we pre-train a customized bilingual BERT, dubbed GigaBERT, designed specifically for Arabic NLP and English-to-Arabic zero-shot transfer learning. We evaluate GigaBERT across four IE tasks: named entity recognition (NER), part-of-speech (POS) tagging, argument role labeling (ARL), and relation extraction (RE). Our best model significantly outperforms mBERT, XLM-RoBERTa, and AraBERT in both supervised and zero-shot transfer settings.

TL;DR

Researchers have introduced GigaBERT, a customized bilingual Transformer model specifically engineered to bridge the gap in Arabic Information Extraction (IE). By combining massive newswire datasets with a novel code-switching pre-training strategy, GigaBERT significantly outperforms existing multilingual models like mBERT and XLM-RoBERTa, as well as the state-of-the-art monolingual AraBERT, in both supervised and zero-shot transfer settings.

What Problem Does This Paper Address?

Information Extraction (IE)—tasks like identifying names (NER), parts of speech (POS), and relationships between entities—is a cornerstone of NLP. While English IE is highly mature, Arabic presents unique challenges: it is morphologically rich, uses a different script, and lacks the vast amounts of labeled data available for English.

Current solutions usually fall into two camps:

  1. Monolingual models (e.g., AraBERT): These are effective for Arabic but cannot leverage English data for transfer learning.
  2. Massively multilingual models (e.g., mBERT, XLM-R): These support 100+ languages but often "dilute" their capacity, leading to sub-optimal performance on specific language pairs like English-Arabic.

The authors address the "zero-shot" challenge: How can we train a model on English data and have it perform accurately on Arabic text without seeing a single Arabic label?

Methods

The authors developed several versions of GigaBERT (v0 through v4) based on the BERT-base architecture. Their approach rests on three pillars:

  • Domain-Specific Pre-training: Unlike mBERT (trained on Wikipedia), GigaBERT is trained heavily on the Gigaword corpus, which contains billions of tokens of newswire text. This aligns the model's knowledge with the formal language used in IE benchmarks.
  • Customized Vocabulary: The team designed a 50k-entry vocabulary. While mBERT only allocates about 5k subwords to Arabic, GigaBERT provides a much finer-grained representation, crucial for handling Arabic’s complex word structures.
  • Synthetic Code-Switching: To force the model to align English and Arabic meanings, they created "code-switched" data. They took English and Arabic sentences and randomly replaced words with their translations from bilingual dictionaries. This creates "anchor points" that help the Transformer project both languages into the same semantic space.

Key Results

GigaBERT-v4 emerged as the clear winner across all evaluated tasks, demonstrating superior performance in both monolingual and cross-lingual settings.

Performance comparison of GigaBERT against baseline models across four IE tasks.
  • Zero-Shot Transfer (English → Arabic): On NER, GigaBERT-v4 achieved an $F_1$ score of 51.5, significantly higher than mBERT (31.1) and XLM-R$_{base}$ (43.5).
  • Supervised Arabic Performance: Even when trained directly on Arabic data, GigaBERT-v4 (84.1 $F_1$ on NER) outperformed the specialized monolingual AraBERT (78.6 $F_1$).
  • Relation Extraction: In the challenging zero-shot RE task, GigaBERT-v4 reached 48.2 $F_1$, nearly 18 points higher than mBERT.

Key Insights and Ideas

  • Anchor Points over Shared Scripts: Conventional wisdom suggested that cross-lingual transfer works best when languages share an alphabet. GigaBERT proves that even with zero shared scripts, a model can achieve high transfer performance if it is forced to align concepts via code-switching during pre-training.
  • Vocabulary Density Matters: By increasing the Arabic-specific vocabulary, the model avoids over-fragmenting Arabic words, allowing it to capture better morphological nuances.
  • Embedding Space Alignment: Analysis showed that GigaBERT is much better at "clustering" parallel English-Arabic sentences together in its internal vector space compared to mBERT, which explains its superior zero-shot performance.
Cosine similarity of bitext vs. random sentence pairs across layers, showing GigaBERT's superior cross-lingual alignment compared to mBERT.

Limitations and Future Work

The authors noted that their domain-adapted XLM-R (GigaXLM-R) showed promise but was limited by computational resources. Future work could involve scaling these bilingual techniques to even larger architectures (like BERT-large) and exploring whether these code-switching "anchor points" can improve transfer for lower-resource languages that lack massive corpora.

Summary updated

Abstract

Multilingual pre-trained Transformers, such as mBERT (Devlin et al., 2019) and XLM-RoBERTa (Conneau et al., 2020a), have been shown to enable the effective cross-lingual zero-shot transfer. However, their performance on Arabic information extraction (IE) tasks is not very well studied. In this paper, we pre-train a customized bilingual BERT, dubbed GigaBERT, that is designed specifically for Arabic NLP and English-to-Arabic zero-shot transfer learning. We study GigaBERT's effectiveness on zero-short transfer across four IE tasks: named entity recognition, part-of-speech tagging, argument role labeling, and relation extraction. Our best model significantly outperforms mBERT, XLM-RoBERTa, and AraBERT (Antoun et al., 2020) in both the supervised and zero-shot transfer settings. We have made our pre-trained models publicly available at https://github.com/lanwuwei/GigaBERT.
BibTeX
@inproceedings{lan2020empirical,
  author = {Wuwei Lan and Chen Yang and Wei Xu and Alan Ritter},
  title = {{An Empirical Study of Pre-trained Transformers for Arabic Information Extraction}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2020},
  doi = {10.18653/v1/2020.emnlp-main.382},
}
Figure