TL;DR
This paper introduces FaBERT, a new BERT-based language model pre-trained specifically on a large corpus of Persian blogs. FaBERT demonstrates improved performance across a variety of Natural Language Understanding (NLU) tasks, highlighting the value of diverse, informal text for training robust language models in Persian.
What Problem Does This Paper Address?
While sophisticated language models like BERT have revolutionized NLP, their application to less-resourced languages like Persian still faces challenges. Existing Persian NLP models often rely on formal text or a mix of sources, potentially missing the nuances of informal language prevalent in everyday communication. This gap is particularly evident in understanding colloquialisms, evolving slang, and diverse linguistic styles found in user-generated content like blogs. Furthermore, large language models (LLMs), while powerful, come with significant computational costs and latency, making smaller, efficient models like BERT crucial for many practical applications. This work aims to bridge this gap by creating a BERT model optimized for Persian, specifically by leveraging the rich, informal data found in Persian blogs.
Methods
The core of this work is the pre-training of a BERT-base model, named FaBERT, using the HmBlogs corpus. This corpus is a collection of 20 million Persian blog posts spanning 15 years, offering a diverse mix of formal and informal texts.
Key methodological aspects include:
- Corpus Pre-processing: The HmBlogs corpus underwent significant cleaning to filter out non-Persian posts, correct typographical errors, and standardize non-standard characters that can cause issues in NLP tasks.
- Tokenizer Adaptation: FaBERT utilizes a WordPiece tokenizer, adapted to handle Persian's unique feature of half-spaces, ensuring accurate text representation. The vocabulary size was set to 50,000 tokens to balance linguistic detail with computational efficiency.
- Pre-training Objectives: FaBERT was pre-trained using Masked Language Modeling (MLM) with a 15% masking rate and whole-word masking for enhanced performance. The Next Sentence Prediction (NSP) task was omitted, as prior research indicated it had no significant positive impact.
- Model Architecture: FaBERT adheres to the standard BERT-base architecture, featuring 12 hidden layers and 12 self-attention heads, making it a compact model with 124 million parameters.
Datasets
FaBERT was pre-trained on the HmBlogs corpus.
For evaluation, FaBERT was fine-tuned and tested on a diverse set of 12 downstream tasks across several datasets:
- Natural Language Inference (NLI) & Question Paraphrasing: FarsTail, SBU-NLI, ParsiNLU Question Paraphrasing.
- Named Entity Recognition (NER): ParsTwiNER (informal tweets), PEYMA (formal news), MultiCoNER v2 (multilingual, fine-grained entities).
- Sentiment Analysis & Irony Detection: DeepSentiPers, MirasOpinion, MirasIrony.
- Question Answering: ParsiNLU Reading Comprehension, PQuAD, PCoQA (conversational QA).
Key Insights and Ideas
- Informal Text is Crucial: Pre-training on a diverse corpus like HmBlogs, rich in informal Persian text, significantly boosts model performance on various NLU tasks. This highlights that neglecting informal language can lead to suboptimal results, even for sophisticated models.
- Compact Models Still Reign: Despite the rise of LLMs, BERT-based models like FaBERT remain highly effective and computationally efficient for many tasks, especially in resource-constrained environments. Their smaller size and faster inference times make them practical for real-world applications.
- Tokenizer Matters: The choice of tokenizer and its vocabulary size impacts how text is processed. FaBERT's tokenizer, while smaller than some others, achieves good compression and effectively handles Persian-specific linguistic features.
- Monolingual Models Outperform Multilingual: The paper reinforces the finding that monolingual BERT models often outperform multilingual counterparts on tasks within their specific language, even when the multilingual models have broader coverage.
Key Results
FaBERT demonstrated strong performance across the evaluated tasks:
- Overall Improvement: FaBERT achieved state-of-the-art or near state-of-the-art results on 9 out of 12 evaluated datasets.
- NLI & Question Paraphrasing: FaBERT showed a +1% improvement in F1 for FarsTail and a +2.88% F1 score for the informal ParsiNLU Question Paraphrasing dataset.
- NER: FaBERT achieved a +1.09% increase in F1 score for the informal ParsTwiNER dataset and performed comparably to ParsBERT on MultiCoNER v2.
- Sentiment Analysis: FaBERT achieved the highest scores in sentiment analysis for both DeepSentiPers and MirasOpinion.
- Question Answering: FaBERT scored +6.24% higher in F1 for the ParsiNLU Reading Comprehension task and showed comparable F1 performance to XLM-R on PQuAD.
- Efficiency: FaBERT is a compact model with 124 million parameters, making it more efficient than many other Persian and multilingual BERT models.
Limitations and Future Work
- Inherent Biases: As FaBERT is trained exclusively on blog data, it inherits potential demographic and sociolinguistic biases present in Persian online communities.
- Sequence Length Limitation: Like other BERT models, FaBERT is limited by the standard 512-token sequence length, which can hinder its ability to process very long documents or capture long-range dependencies. The authors suggest that advancements in long-context transformer models could be adapted for Persian.
- Dataset Scarcity: The Persian NLP landscape suffers from a lack of datasets and benchmarks for training and evaluating text embeddings. Future work should focus on creating resources for Persian sentence embeddings to advance applications like dense retrieval and semantic search.
The authors suggest that their approach, combining BERT's efficiency with rich, diverse training data, positions FaBERT as a valuable tool for Persian NLP, especially in resource-constrained environments. They also highlight the potential for using similar diverse corpora for pre-training other language models.
Summary updated