Introduction
The rapid advancement of Transformer-based models has revolutionized Natural Language Processing (NLP), yet this progress is unevenly distributed. High-resource languages benefit from massive datasets and immense computational power, while low-resource languages (LRLs) like Assamese—spoken by approximately 23 million people in Eastern India—are often left behind. Existing multilingual models, such as mBERT or MuRIL, frequently struggle to capture the unique morphological and phonological characteristics of Assamese.
Furthermore, models trained on limited data often suffer from anisotropy, a phenomenon where word embeddings cluster within a narrow cone in the vector space. This "clumping" makes it difficult for the model to distinguish between semantically distinct candidates, particularly in tasks like multiple-choice question answering or named entity recognition. AxomiyaBERTa was developed to bridge this gap by prioritizing linguistic awareness and architectural efficiency over sheer scale.
Methodology
AxomiyaBERTa utilizes an ALBERT (A Lite BERT) architecture, chosen for its parameter efficiency. The researchers introduced several methodological innovations to optimize performance in a resource-constrained environment.
Masked Language Modeling (MLM) and Efficiency
Standard BERT pretraining involves both Masked Language Modeling (MLM) and Next Sentence Prediction (NSP). However, the authors found that for LRLs, the computational cost of NSP does not yield significant performance improvements. Consequently, AxomiyaBERTa was trained exclusively on MLM, allowing the model to be pretrained on a single NVIDIA A100 GPU in just 72 hours.
The Embedding Disperser
To combat the issue of anisotropy, the model incorporates a novel "Embedding Disperser" during the fine-tuning stage. This module uses a combined loss function—Binary Cross Entropy (BCE) and Cosine loss—to "push" embeddings apart, creating a more isotropic and uniform vector space. This dispersion is vital for tasks requiring precise discrimination between multiple options.
Phonological Attention
A standout feature of AxomiyaBERTa is its integration of phonological information. Using the PanPhon library, the model extracts 24 sub-segmental articulatory features, such as voicing and place of articulation. By treating the "sound" of a word as a semantic signal, the model can better disambiguate morphologically complex terms and distinguish between similar-sounding entities, such as person names versus organization names.
Experimental Setup and Datasets
The model was pretrained on a corpus of approximately 26 million tokens aggregated from four primary sources: Assamese Wikidumps, the OSCAR corpus, PMIndia parallel data, and CC-100. For evaluation, the researchers utilized benchmarks including IndicGLUE (for Cloze-style QA), AsNER, and a newly adapted Assamese version of the ECB+ corpus (T-ECB+) for cross-document event coreference resolution.
Key Results and Analysis
AxomiyaBERTa consistently outperformed larger multilingual baselines across various Assamese-specific tasks.
Named Entity Recognition (NER)
The model achieved a state-of-the-art (SOTA) 86.90 F1 score on the AsNER dataset. This performance surpasses that of MuRIL, mBERT, and IndicBERT, highlighting the effectiveness of phonological awareness in identifying entities within the Assamese context.
Cloze-QA and Title Prediction
The impact of phonological signals was most pronounced in Wiki Title Prediction, where accuracy jumped from 26% to 59%. The Embedding Disperser also played a crucial role; without it, the model's embeddings remained too tightly packed to effectively solve multiple-choice problems.
Coreference Resolution
On the T-ECB+ task, AxomiyaBERTa outperformed other Transformer baselines on standard metrics like MUC and CoNLL F1. This result is particularly significant because it demonstrates that a model can achieve high performance in long-context reasoning tasks even without explicit NSP training, provided its internal representations are sufficiently robust.
Conclusion
AxomiyaBERTa proves that "Big Tech" compute budgets are not a prerequisite for high-quality NLP in underrepresented languages. By focusing on phonological signals and addressing structural issues like embedding anisotropy, a 66M parameter model can set new benchmarks for LRLs. This work provides a sustainable and linguistically grounded roadmap for future research into low-resource language modeling.
Source: Findings of the Association for Computational Linguistics: ACL 2023, pages 11629–11646.
Summary updated