ArabicTransformer: Efficient Large Arabic Language Model with Funnel Transformer and ELECTRA Objective

Sultan Alrowili, Vijay ShankerEMNLP 2021 · DOI

Introduction

Pre-training large-scale Transformer models like BERT has revolutionized Natural Language Processing (NLP) but remains computationally prohibitive. For languages like Arabic, the high cost of training limits the ability of researchers with modest resources to develop or fine-tune state-of-the-art models. ArabicTransformer addresses this "entry fee" by combining the structural efficiency of the Funnel Transformer with the sample-efficient ELECTRA training objective.

Methodology

The core innovation of ArabicTransformer lies in its architectural approach to reducing "sequential redundancy." Unlike standard Transformers that maintain a constant sequence length throughout the encoder, the Funnel Transformer progressively compresses the sequence of hidden states.

The Funnel Transformer architecture uses pooling to compress sequence length in the encoder, followed by up-sampling in the decoder.

As shown in the architecture diagram, the model uses a "pooling" technique to reduce the number of operations (FLOPs) as the representation goes deeper. A decoder then up-samples the representation back to the original length for pre-training tasks.

To further enhance efficiency, the model employs the ELECTRA training objective. Instead of standard Masked Language Modeling (MLM), ELECTRA uses a discriminator to identify tokens replaced by a small generator. This task is significantly more sample-efficient as the model learns from every token in the sentence, rather than just the masked subset.

Experimental Setup

The authors pre-trained two configurations—B6-6-6 (three blocks of 6 layers) and B4-4-4—on a 45GB corpus of Arabic text. This dataset included the Arabic Wikipedia dump, the Abu El-Khair corpus, and the Unshuffled Arabic Oscar dataset. Evaluation was conducted across several established Arabic benchmarks, including Question Answering (ARCD, TyDi QA) and Sentiment Analysis (HARD, AJGT, ArSarcasm).

Results and Performance

ArabicTransformer achieves state-of-the-art results while maintaining high computational efficiency. In Question Answering tasks, the B6-6-6 model outperformed established models like AraELECTRA and AraBERT.

Comparison of ArabicTransformer (Ours) against baseline models on TyDiQA and ARCD benchmarks, showing SOTA performance in bold.

Beyond performance metrics, the model's efficiency is its standout feature. The B6-6-6 model achieved its peak performance with a "Computational Ratio" of only 0.5x compared to AraELECTRA. In practical terms, the B6-6-6 model reached its performance peak in just 2 days and 10 hours on a TPUv3-32, whereas AraELECTRA required an estimated 24 days on a TPUv3-8.

In sentiment analysis, the model set new bars for the HARD dataset (96.6% accuracy) and the ArSarcasm task (70.8%). Interestingly, the B4-4-4 model was roughly 28% faster to fine-tune than standard base models, making it an ideal choice for researchers with limited hardware.

Conclusion

ArabicTransformer demonstrates that architectural innovations like sequence pooling can break the linear relationship between model scale and computational cost. By achieving SOTA results at approximately 40% of the cost of previous leading models, it provides a more accessible and sustainable path for high-performance Arabic NLP research.

Summary updated

BibTeX
@inproceedings{alrowili2021arabictransformer,
  author = {Sultan Alrowili and Vijay Shanker},
  title = {{ArabicTransformer: Efficient Large Arabic Language Model with Funnel Transformer and ELECTRA Objective}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2021},
  doi = {10.18653/v1/2021.findings-emnlp.108},
}
Figure