AxomiyaBERTa: A Phonologically-aware Transformer Model for Assamese

Abhijnan Nath, Sheikh Mannan, Nikhil Krishnaswamy

Introduction

The rapid advancement of Transformer-based models has revolutionized Natural Language Processing (NLP), yet this progress is unevenly distributed. High-resource languages benefit from massive datasets and immense computational power, while low-resource languages (LRLs) like Assamese—spoken by approximately 23 million people in Eastern India—are often left behind. Existing multilingual models, such as mBERT or MuRIL, frequently struggle to capture the unique morphological and phonological characteristics of Assamese.

Furthermore, models trained on limited data often suffer from anisotropy, a phenomenon where word embeddings cluster within a narrow cone in the vector space. This "clumping" makes it difficult for the model to distinguish between semantically distinct candidates, particularly in tasks like multiple-choice question answering or named entity recognition. AxomiyaBERTa was developed to bridge this gap by prioritizing linguistic awareness and architectural efficiency over sheer scale.

Methodology

AxomiyaBERTa utilizes an ALBERT (A Lite BERT) architecture, chosen for its parameter efficiency. The researchers introduced several methodological innovations to optimize performance in a resource-constrained environment.

Masked Language Modeling (MLM) and Efficiency

Standard BERT pretraining involves both Masked Language Modeling (MLM) and Next Sentence Prediction (NSP). However, the authors found that for LRLs, the computational cost of NSP does not yield significant performance improvements. Consequently, AxomiyaBERTa was trained exclusively on MLM, allowing the model to be pretrained on a single NVIDIA A100 GPU in just 72 hours.

The Embedding Disperser

To combat the issue of anisotropy, the model incorporates a novel "Embedding Disperser" during the fine-tuning stage. This module uses a combined loss function—Binary Cross Entropy (BCE) and Cosine loss—to "push" embeddings apart, creating a more isotropic and uniform vector space. This dispersion is vital for tasks requiring precise discrimination between multiple options.

Figure 1: The AxomiyaBERTa architecture, illustrating the dual-loss training approach (BCE and Cosine loss) used to disperse embeddings for Cloze-QA tasks.

Phonological Attention

A standout feature of AxomiyaBERTa is its integration of phonological information. Using the PanPhon library, the model extracts 24 sub-segmental articulatory features, such as voicing and place of articulation. By treating the "sound" of a word as a semantic signal, the model can better disambiguate morphologically complex terms and distinguish between similar-sounding entities, such as person names versus organization names.

Experimental Setup and Datasets

The model was pretrained on a corpus of approximately 26 million tokens aggregated from four primary sources: Assamese Wikidumps, the OSCAR corpus, PMIndia parallel data, and CC-100. For evaluation, the researchers utilized benchmarks including IndicGLUE (for Cloze-style QA), AsNER, and a newly adapted Assamese version of the ECB+ corpus (T-ECB+) for cross-document event coreference resolution.

Key Results and Analysis

AxomiyaBERTa consistently outperformed larger multilingual baselines across various Assamese-specific tasks.

Named Entity Recognition (NER)

The model achieved a state-of-the-art (SOTA) 86.90 F1 score on the AsNER dataset. This performance surpasses that of MuRIL, mBERT, and IndicBERT, highlighting the effectiveness of phonological awareness in identifying entities within the Assamese context.

Cloze-QA and Title Prediction

The impact of phonological signals was most pronounced in Wiki Title Prediction, where accuracy jumped from 26% to 59%. The Embedding Disperser also played a crucial role; without it, the model's embeddings remained too tightly packed to effectively solve multiple-choice problems.

Figure 2: Density plot of similarity scores, demonstrating how different scoring metrics and dispersion techniques affect the distribution of model outputs.

Coreference Resolution

On the T-ECB+ task, AxomiyaBERTa outperformed other Transformer baselines on standard metrics like MUC and CoNLL F1. This result is particularly significant because it demonstrates that a model can achieve high performance in long-context reasoning tasks even without explicit NSP training, provided its internal representations are sufficiently robust.

Conclusion

AxomiyaBERTa proves that "Big Tech" compute budgets are not a prerequisite for high-quality NLP in underrepresented languages. By focusing on phonological signals and addressing structural issues like embedding anisotropy, a 66M parameter model can set new benchmarks for LRLs. This work provides a sustainable and linguistically grounded roadmap for future research into low-resource language modeling.

Source: Findings of the Association for Computational Linguistics: ACL 2023, pages 11629–11646.

Summary updated

Abstract

Despite their successes in NLP, Transformer-based language models still require extensive computing resources and suffer in low-resource or low-compute settings. In this paper, we present AxomiyaBERTa, a novel BERT model for Assamese, a morphologically-rich low-resource language (LRL) of Eastern India. AxomiyaBERTa is trained only on the masked language modeling (MLM) task, without the typical additional next sentence prediction (NSP) objective, and our results show that in resource-scarce settings for very low-resource languages like Assamese, MLM alone can be successfully leveraged for a range of tasks. AxomiyaBERTa achieves SOTA on token-level tasks like Named Entity Recognition and also performs well on "longer-context" tasks like Cloze-style QA and Wiki Title Prediction, with the assistance of a novel embedding disperser and phonological signals respectively. Moreover, we show that AxomiyaBERTa can leverage phonological signals for even more challenging tasks, such as a novel cross-document coreference task on a translated version of the ECB+ corpus, where we present a new SOTA result for an LRL. Our source code and evaluation scripts may be found at https://github.com/csu-signal/axomiyaberta.
BibTeX
@inproceedings{nath2023axomiyaberta,
  author = {Abhijnan Nath and Sheikh Mannan and Nikhil Krishnaswamy},
  title = {{AxomiyaBERTa: A Phonologically-aware Transformer Model for Assamese}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
  doi = {10.18653/v1/2023.findings-acl.739},
}
Figure