Abstract
Arabic is a morphological rich language, posing many challenges for information extraction (IE) tasks, including Named Entity Recognition (NER), Part-of-Speech tagging (POS), Argument Role Labeling (ARL), and Relation Extraction (RE). A few multilingual pre-trained models have been proposed and show good performance for Arabic, however, most experiment results are reported on language understanding tasks, such as natural language inference, question answering and sentiment analysis. Their performance on the IE tasks is less known, in particular, the cross-lingual transfer capability from English to Arabic. In this work, we pre-train a Gigaword-based bilingual language model (GigaBERT) to study these two distant languages as well as zero-short transfer learning on various IE tasks. Our GigaBERT outperforms multilingual BERT and monolingual AraBERT on these tasks, in both supervised and zero-shot learning settings.
Introduction
Arabic presents unique challenges for Natural Language Processing due to its complex morphology and a script entirely distinct from English. While massive multilingual models like mBERT and XLM-R support over 100 languages, their performance often falters on structured Information Extraction (IE) tasks—such as identifying names (NER) or relationships (RE)—especially in zero-shot cross-lingual transfer scenarios. Most existing research focuses on high-level language understanding (e.g., sentiment analysis), leaving the performance of these models on structural IE tasks under-explored. Furthermore, general-purpose models often lack the specific vocabulary and in-domain data required to master Arabic's complex structure.
Methodology
The researchers developed GigaBERT, a model based on the standard BERT-base architecture (12 layers, 768 hidden dimensions) optimized specifically for the English-Arabic pair. Key methodological choices include:
- Bilingual Pre-training: Instead of 100+ languages, the model focuses on two, utilizing the Gigaword corpora (newswire text) for both English and Arabic.
- Vocabulary Customization: A specific 50k-entry vocabulary was designed. While mBERT and XLM-R have few Arabic-specific tokens, GigaBERT allocates roughly half its vocabulary to each language, ensuring superior sub-word representation.
- Iterative Optimization: Four versions (v0 to v3) were released, experimenting with tokenization methods (SentencePiece vs. WordPiece), casing, and additional data sources like Wikipedia and the Oscar dataset.
Key Results
GigaBERT-v3 emerged as the top performer across all evaluated IE tasks. In supervised settings, GigaBERT-v3 achieved an F1 score of 83.1 on Arabic NER, significantly outperforming the state-of-the-art monolingual AraBERT (78.6) and mBERT (72.9).
In the challenging zero-shot transfer setting (English → Arabic), GigaBERT-v1 achieved an F1 of 49.1 for NER, nearly 20 points higher than mBERT (30.8). For Relation Extraction, GigaBERT-v3 outperformed all other models in both supervised (84.3 F1) and zero-shot (48.2 F1) settings.
Discussion and Insights
Several factors contribute to GigaBERT's success:
- The Vocabulary Gap: General multilingual models are spread too thin. Dedicating more "vocabulary real estate" to Arabic sub-words allows GigaBERT to capture the language's morphology more effectively.
- In-Domain Data: Pre-training on newswire (Gigaword) for tasks evaluated on newswire (ACE 2005) provides a massive boost compared to models trained primarily on Wikipedia or web crawls.
- Balanced Embedding Space: GigaBERT balances the representation of both languages better than mBERT. It maintains enough distinction to perform well on downstream tasks while enabling effective cross-lingual transfer.
- Transfer Without Shared Scripts: GigaBERT achieves strong zero-shot transfer despite English and Arabic sharing almost no sub-words or script, suggesting the Transformer encoder successfully projects distant languages into a shared conceptual space.
Conclusion and Future Work
While GigaBERT significantly narrows the gap, zero-shot performance still lags behind supervised performance. Future work could explore how bilingual optimization affects other areas like machine translation or generative tasks. Additionally, the researchers noted that continuing pre-training with longer sequence lengths was vital for performance on complex extraction tasks, suggesting that standard 128-token limits are a bottleneck for IE.
Summary updated