Abstract
This work presents BanglaNLG, a comprehensive benchmark for evaluating natural language generation (NLG) models in Bangla, a widely spoken yet low-resource language. We aggregate six challenging conditional text generation tasks under the BanglaNLG benchmark, introducing a new dataset on dialogue generation in the process. Furthermore, using a clean corpus of 27.5 GB of Bangla data, we pretrain BanglaT5, a sequence-to-sequence Transformer language model for Bangla. BanglaT5 achieves state-of-the-art performance in all of these tasks, outperforming several multilingual models by up to 9% absolute gain and 32% relative gain. We are making the new dialogue dataset and the BanglaT5 model publicly available at https://github.com/csebuetnlp/BanglaNLG in the hope of advancing future research on Bangla NLG.
Introduction
Despite being the sixth most spoken language globally with over 230 million native speakers, Bangla remains a "low-resource" language in the context of Natural Language Processing (NLP). While recent breakthroughs in Large Language Models (LLMs) have transformed the field, research for Bangla has historically been fragmented, lacking a unified benchmark to evaluate generative capabilities across diverse contexts—from news summarization to conversational dialogue.
Furthermore, existing multilingual models often underperform on Bangla because the language-specific portion of their training data is frequently noisy or insufficient. To address these gaps, we introduce BanglaNLG, a comprehensive benchmark for Bangla NLG, and BanglaT5, a specialized sequence-to-sequence model designed to serve as a high-performance engine for the language.
The BanglaNLG Benchmark
The BanglaNLG benchmark aggregates six challenging conditional text generation tasks to provide a holistic evaluation of model performance:
- Machine Translation (MT): Evaluating Bangla $\leftrightarrow$ English translation using BanglaNMT and FLoRes-100.
- Text Summarization (TS): Abstractive summarization based on the Bangla portion of the XL-Sum dataset.
- Question Answering (QA): Utilizing BQA, a combination of translated SQuAD 2.0 and TyDi-QA.
- Multi-turn Dialogue (MTD): A newly introduced dataset created by translating the DailyDialog dataset into Bangla, followed by manual verification of the evaluation sets.
- News Headline Generation (NHG): Generating concise titles from news articles (XL-Sum).
- Cross-lingual Summarization (XLS): Summarizing English articles into Bangla using the CrossSum dataset.
BanglaT5: Architecture and Pretraining
BanglaT5 is a Transformer-based encoder-decoder model following the T5 (Text-to-Text Transfer Transformer) architecture. Rather than relying on sheer model scale, the development of BanglaT5 focused on data quality and language-specific optimization.
Key Technical Features
- Pretraining Objective: The model employs a "span-correction" objective, where it learns to reconstruct missing segments of text. This method is particularly effective for generative tasks.
- Clean Data Corpus: Instead of using massive but noisy datasets like mC4, BanglaT5 was pretrained on Bangla2B+, a meticulously filtered 27.5 GB corpus of clean Bangla text.
- Vocabulary Optimization: We trained a 32k subword vocabulary with high character coverage (0.99995). This ensures the model handles the nuances of the Bangla script efficiently while retaining approximately 10% non-Bangla tokens to manage code-switching and technical terminology.
Key Results and Insights
BanglaT5 demonstrates that specialized monolingual models can significantly outperform much larger multilingual counterparts when trained on high-quality data.
Performance Gains
BanglaT5 achieved an average 4% absolute gain over the next best model (mT5) across all benchmark tasks. The most significant improvement was observed in Question Answering, where BanglaT5 outperformed competitors by up to 9.54% in F1 score. The model established new state-of-the-art (SOTA) records for Bangla on three of the six benchmark tasks.
Efficiency and Speed
With 247 million parameters, BanglaT5 is more compact than multilingual giants like mBART-50 or XLM-ProphetNet (which exceed 600 million parameters). This specialization results in 2x to 2.5x faster training and inference times, making it a more sustainable and accessible choice for researchers.
The "Clean Data" Advantage
By avoiding noisy web-scraped data that often contains offensive or garbled text, BanglaT5 exhibits greater robustness and better generalization. Interestingly, despite its monolingual focus, the model showed surprising strength in Machine Translation. This suggests that the presence of foreign tokens (e.g., English names or technical terms) in the "clean" Bangla corpus allows the model to develop implicit cross-lingual transfer capabilities during pretraining.
Limitations and Future Directions
While BanglaT5 excels in formal text generation, it lagged slightly behind XLM-ProphetNet in the Multi-turn Dialogue task. This is likely because the pretraining data lacked colloquial or conversational Bangla, which differs significantly from formal written prose. Additionally, performance on cross-lingual summarization remains a challenge for all models due to the scarcity of training data (approximately 1,200 examples).
Future work will involve expanding the BanglaNLG benchmark to include personalized dialogue and conversational question answering. We also aim to investigate the impact of incorporating more diverse, colloquial datasets to improve performance in informal contexts.
Conclusion
BanglaNLG and BanglaT5 represent a significant step forward for Bangla NLP. By providing both a standardized evaluation framework and a high-efficiency pretrained model, we hope to accelerate research and development for this widely spoken yet underserved language. The dataset and model are publicly available to the research community to foster further innovation.
Summary updated