AssameseLM-E2B: An Open-Weight Assamese Language Model for Efficient Consumer-Grade Deployment

Pragyan PhukanarXiv 2026 · arXiv

TL;DR

AssameseLM-E2B is a 2-billion-parameter model specifically optimized for the Assamese language, achieving a 5x improvement in tokenization efficiency over general-purpose models like Llama 3.2. By combining vocabulary expansion with a "Hybrid LoRA" training approach, the authors created a lightweight model that rivals 3B-parameter baselines in linguistic accuracy while remaining deployable on consumer-grade hardware.

What Problem Does This Paper Address?

Large Language Models (LLMs) typically struggle with low-resource languages like Assamese due to three main factors:

  1. The "Token Tax": Most tokenizers are trained on Latin-centric data. For Assamese, they often fall back to byte-level encoding, splitting a single character into 3–4 tokens. This bloats sequence lengths, slows down inference, and exhausts context windows prematurely.
  2. Cross-Lingual Contamination: Assamese shares a script with Bengali. Because Bengali has a larger digital footprint, general models often "hallucinate" Bengali vocabulary or grammar when prompted in Assamese.
  3. Receptive-Productive Decoupling: The authors identify a phenomenon where models understand an Assamese query (receptive) but lack the specific decoding weights to generate a grammatically correct Assamese response (productive), often resulting in "generative collapse."

Methods

The researchers built AssameseLM-E2B on the Gemma-2B architecture using a multi-stage adaptation pipeline:

  • Vocabulary Expansion: They added 20,000 native Assamese tokens using a SentencePiece BPE tokenizer, specifically filtered to exclude orthographically invalid subwords (like isolated vowel signs).
  • Hybrid LoRA: Standard Low-Rank Adaptation (LoRA) only trains adapter weights. To accommodate the new vocabulary, the authors implemented "Hybrid LoRA," which explicitly unfreezes the input embedding layer and the final classification head while using LoRA for the attention and feed-forward layers.
  • Continual Pre-Training (CPT): The model was trained on 0.5 billion Assamese tokens to learn syntax and semantics.
  • Supervised Fine-Tuning (SFT): The instruction-tuned variant (AssameseLM-E2B-it) was trained on ~30k instruction-response pairs across domains like RAG, direct chat, and translation.
Training dynamics during the Continual Pre-Training phase, showing stable convergence of loss and perplexity.

Datasets

  • CPT Corpus: A hybrid of clean web-scrapes (Common Crawl, Wikipedia, Sangraha) and a high-fidelity OCR Literary Corpus consisting of 100 digitized Assamese books to provide formal grammatical anchors.
  • SFT Dataset: 29,319 pairs, including translated subsets of Alpaca and OpenOrca, alongside 8,000 specialized Retrieval-Augmented Generation (RAG) pairs.
  • Evaluation: The model was tested against the FLORES-200 Assamese split and a custom 150-prompt multi-domain zero-shot suite.

Key Insights and Ideas

  • Efficiency is King: By reducing the "Token Tax," the model can process three times more information in its context window compared to base multilingual models.
  • The "Hourglass" Bias: During SFT evaluation, the model showed a bimodal distribution in factuality. It either answered perfectly from memory or triggered a "refusal" template (a side effect of heavy RAG training). This highlights the difficulty of balancing groundedness with open-domain recall.
  • Script Invariance: The paper emphasizes the need for Phonetic Transliteration Data Augmentation. Users often type Assamese using Romanized Latin script; models must be trained on dual-script pairs to remain robust.

Key Results

  • Tokenization: AssameseLM-E2B required only 35,107 tokens to encode a test corpus that Llama-3.2-3B needed 166,218 tokens for—a nearly 500% reduction in overhead.
  • Predictive Power: It achieved a Bits-per-Byte (BPB) score of 0.7488 on FLORES-200, outperforming Qwen2.5-3B (0.9042) and matching the much larger Llama-3.2-3B.
  • RAG Performance: The model achieved a 96.7% overall RAG score, with a perfect 100% refusal accuracy on adversarial "unanswerable" prompts, whereas baseline models (Gemma, Llama, Qwen) failed to abstain and instead hallucinated answers.
Comparison of RAG performance across models, showing AssameseLM's dominance in grounded generation.

Limitations and Future Work

  • RAG-Prior Refusal: The model occasionally refuses to answer questions it actually knows the answer to if no external context is provided, due to its strict RAG training.
  • Parameter Scale: While 2B is efficient, it limits complex reasoning compared to 7B+ models. The authors plan to scale this methodology to larger architectures (7B–9B) in the future.
  • Context Window: Future work will focus on expanding the context window to support full-document summarization and multimodal historical manuscript transcription.

Summary updated

Abstract

While Large Language Models (LLMs) excel in high-resource languages, their performance often degrades in low-resource languages due to tokenization inefficiencies, generative instability, and linguistic interference. This paper presents AssameseLM-E2B and its instruction-tuned variant, AssameseLM-E2B-it, a 2-billion-parameter model adapted specifically for low resource Assamese language. Built upon the Gemma-4 architecture, the model incorporates an expanded byte-level BPE vocabulary to reduce tokenization overhead by nearly 500% compared with multilingual 2B--3B baselines. We employ a parameter-efficient Continual Pre-Training (CPT) pipeline with Hybrid LoRA, followed by Supervised Fine-Tuning (SFT) on a balanced, multi-domain dataset to enhance instruction-following and Retrieval-Augmented Generation (RAG) capabilities. Evaluation demonstrates a Bits-per-Byte (BPB) score of 0.7488 on FLORES-200, alongside an overall RAG score of 96.7% and 100% refusal accuracy on adversarial unanswerable prompts. Comparative analysis reveals significant limitations in general-purpose baselines, including receptive-productive decoupling, generative degradation on Romanized Assamese, and unintended Bengali script assimilation. These results demonstrate the potential of parameter-efficient adaptation for developing capable, locally deployable language models for low-resource languages under constrained computational resources.
BibTeX
@misc{phukan2026assameselm,
  author = {Pragyan Phukan},
  title = {{AssameseLM-E2B: An Open-Weight Assamese Language Model for Efficient Consumer-Grade Deployment}},
  year = {2026},
  eprint = {2611.06170},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.06170},
}
Figure