Abstract
Multilingual pre-trained Transformers, such as mBERT and XLM-RoBERTa, have enabled effective cross-lingual zero-shot transfer. However, their performance on Arabic information extraction (IE) tasks is not well studied. We introduce GigaBERT, a customized bilingual BERT designed specifically for Arabic NLP and English-to-Arabic zero-shot transfer. We evaluate GigaBERT on four IE tasks: named entity recognition (NER), part-of-speech (POS) tagging, argument role labeling (ARL), and relation extraction (RE). GigaBERT significantly outperforms mBERT, XLM-RoBERTa, and AraBERT in both supervised and zero-shot settings.
Introduction
Multilingual Transformers have demonstrated impressive zero-shot cross-lingual transfer capabilities, allowing models fine-tuned on English data to be evaluated on Arabic without target-language training examples. However, Arabic IE performance often lags behind supervised models. English and Arabic present a particular challenge due to their entirely different scripts, which limits natural "anchor points" (like shared subwords) between the languages. This study investigates whether a dedicated, domain-specific bilingual model can bridge this gap more effectively than massive multilingual models.
Methodology
GigaBERT utilizes a BERT-base architecture pre-trained from scratch on English and Arabic text. The approach diverges from standard multilingual pre-training in three key ways:
- Custom Unified Vocabulary: Instead of the skewed vocabularies of mBERT or XLM-R, GigaBERT uses a 50k WordPiece vocabulary specifically balanced for English and Arabic.
- Code-Switched Data Augmentation: To force the model to learn cross-lingual alignments across different scripts, synthetic code-switched data was generated. Using bilingual dictionaries, up to 30% of words in a sentence were randomly replaced with their translations.
- Domain Adaptation: Pre-training focused heavily on the newswire domain using Gigaword (v5), Wikipedia, and the Arabic Oscar corpus.
Results and Discussion
GigaBERT-v4 achieves state-of-the-art results in zero-shot transfer from English to Arabic. As shown in the performance comparison below, GigaBERT-v4 significantly outperforms mBERT and XLM-R across most IE tasks.
Key Insights
- Bilingual Advantage: A carefully tuned bilingual model with a balanced vocabulary can outperform massive models trained on 100+ languages for specific, challenging language pairs.
- Impact of Code-Switching: Injecting synthetic code-switched text during pre-training creates artificial "anchor points" that dramatically improve zero-shot transfer. Comparing GigaBERT-v4 to the non-code-switched v3 shows a significant jump in zero-shot ARL (from 51.0 to 56.1 F1) and RE (from 46.8 to 48.2 F1).
- Semantic Alignment: Analysis of the embedding space shows that GigaBERT learns stronger language-agnostic representations. As illustrated in the figure below, GigaBERT successfully clusters parallel English-Arabic sentences (bitext) together while keeping random pairs distinct, a task where mBERT and XLM-R show less separation.
Conclusion
GigaBERT demonstrates that domain-specific bilingual pre-training and synthetic code-switching are highly effective for cross-lingual transfer in challenging language pairs like English-Arabic. The model not only excels in zero-shot transfer but also outperforms dedicated monolingual models like AraBERT in supervised Arabic tasks.
Summary updated