Abstract
AxomiyaBERTa is a specialized Transformer model designed for Assamese, a morphologically rich but low-resource language. By integrating phonological (sound-based) features and a novel "embedding disperser" to combat representation decay, the model achieves state-of-the-art results on Assamese Named Entity Recognition and introduces the first-ever event coreference resolution benchmark for the language. This research demonstrates that for low-resource languages, architectural efficiency and sound-based signals can be more effective than simply increasing model scale.
Introduction
Despite the rapid advancement of Transformer-based models, most state-of-the-art NLP systems remain "data-hungry" giants trained primarily on high-resource languages like English. For languages like Assamese—spoken by millions in Eastern India but significantly underrepresented in digital corpora—standard models often fail to capture the linguistic nuances necessary for high-level performance.
This paper identifies two major hurdles in developing effective models for Assamese:
- Data Scarcity: Assamese lacks the massive, high-quality datasets required to train standard BERT-like models effectively.
- Representation Degeneration (Anisotropy): In low-resource or low-compute settings, Transformer embeddings tend to cluster into a narrow "cone" in vector space. This anisotropy makes it difficult for the model to distinguish between different words or concepts, as their vector representations become too similar to differentiate.
Methodology
Model Architecture
The authors developed AxomiyaBERTa, a 66-million parameter model based on the ALBERT (A Lite BERT) architecture. To optimize for limited compute and data, the model was trained exclusively on the Masked Language Modeling (MLM) task, omitting the Next Sentence Prediction (NSP) objective commonly found in standard BERT training.
The Embedding Disperser
To address the problem of anisotropy, the researchers introduced an "Embedding Disperser." This fine-tuning technique utilizes a combination of Binary Cross Entropy (BCE) and Cosine Embedding loss. By mathematically "pushing" word vectors apart, this component creates a more isotropic (uniform) embedding space, significantly improving the model's discriminatory power in downstream tasks.
Phonological and Articulatory Attention
Assamese possesses unique sound patterns that are often lost in standard text-based tokenization. To capture these, the authors converted text into International Phonetic Alphabet (IPA) transcriptions and extracted 24 subsegmental features (such as "voicing" or "place of articulation") using the PanPhon library. These sound signals were integrated into the model alongside traditional text embeddings, providing vital context when semantic data was sparse.
Datasets and Evaluation
The model was pretrained on a collection of five publicly available Assamese datasets, totaling approximately 26 million tokens, including Assamese Wikidumps, OSCAR, PMIndia, and CC-100. Additionally, the authors introduced T-ECB+, a version of the Event Coreference Bank+ (ECB+) corpus translated into Assamese.
Evaluation was conducted across several benchmarks:
- IndicGLUE: For Cloze-style QA and Wiki Title Prediction.
- WikiNER and AsNER: For Named Entity Recognition.
- T-ECB+: For event coreference resolution.
Key Results and Insights
State-of-the-Art Performance
AxomiyaBERTa achieved a State-of-the-Art (SOTA) F1 score of 86.90 on the AsNER dataset. This significantly outperformed much larger multilingual models, including MuRIL (80.69) and IndicBERT (68.37). These results suggest that language-specific optimization can outweigh the benefits of massive multilingual pretraining.
The Impact of Phonology
The inclusion of phonological signals proved to be a transformative addition. In Wikipedia Section Title Prediction, accuracy jumped from 26.19% to 59.26% when sound-based features were added. Phonology acts as a powerful semantic proxy; for instance, the sound of a word can help a model distinguish between a person's name and an organization named after that person, even when the orthography (spelling) is identical.
Coreference and Isotropy
On the T-ECB+ task, the model outperformed lemma-based baselines and other Transformers. The phonological signals helped the model correlate synonyms and loanwords that lack surface-level similarity but share phonetic characteristics. Furthermore, the embedding disperser improved performance on multiple-choice tasks where candidate answers were semantically similar, proving that managing the geometry of the embedding space is as crucial as the volume of training data.
Limitations and Future Work
While AxomiyaBERTa represents a significant step forward, it is currently limited to the Assamese language and may not generalize to other scripts without retraining. Additionally, because the training data is sourced from internet corpora, the model may exhibit bias toward formal language and struggle with regional dialects.
Future research will focus on incorporating phonological signals directly into the pretraining phase rather than just fine-tuning, and exploring contrastive learning frameworks to further refine event coreference resolution.
Summary updated