TL;DR
AssameseLM-E2B is a 2-billion-parameter model specifically optimized for the Assamese language, achieving a 5x improvement in tokenization efficiency over general-purpose models like Llama 3.2. By combining vocabulary expansion with a "Hybrid LoRA" training approach, the authors created a lightweight model that rivals 3B-parameter baselines in linguistic accuracy while remaining deployable on consumer-grade hardware.
What Problem Does This Paper Address?
Large Language Models (LLMs) typically struggle with low-resource languages like Assamese due to three main factors:
- The "Token Tax": Most tokenizers are trained on Latin-centric data. For Assamese, they often fall back to byte-level encoding, splitting a single character into 3–4 tokens. This bloats sequence lengths, slows down inference, and exhausts context windows prematurely.
- Cross-Lingual Contamination: Assamese shares a script with Bengali. Because Bengali has a larger digital footprint, general models often "hallucinate" Bengali vocabulary or grammar when prompted in Assamese.
- Receptive-Productive Decoupling: The authors identify a phenomenon where models understand an Assamese query (receptive) but lack the specific decoding weights to generate a grammatically correct Assamese response (productive), often resulting in "generative collapse."
Methods
The researchers built AssameseLM-E2B on the Gemma-2B architecture using a multi-stage adaptation pipeline:
- Vocabulary Expansion: They added 20,000 native Assamese tokens using a SentencePiece BPE tokenizer, specifically filtered to exclude orthographically invalid subwords (like isolated vowel signs).
- Hybrid LoRA: Standard Low-Rank Adaptation (LoRA) only trains adapter weights. To accommodate the new vocabulary, the authors implemented "Hybrid LoRA," which explicitly unfreezes the input embedding layer and the final classification head while using LoRA for the attention and feed-forward layers.
- Continual Pre-Training (CPT): The model was trained on 0.5 billion Assamese tokens to learn syntax and semantics.
- Supervised Fine-Tuning (SFT): The instruction-tuned variant (AssameseLM-E2B-it) was trained on ~30k instruction-response pairs across domains like RAG, direct chat, and translation.
Datasets
- CPT Corpus: A hybrid of clean web-scrapes (Common Crawl, Wikipedia, Sangraha) and a high-fidelity OCR Literary Corpus consisting of 100 digitized Assamese books to provide formal grammatical anchors.
- SFT Dataset: 29,319 pairs, including translated subsets of Alpaca and OpenOrca, alongside 8,000 specialized Retrieval-Augmented Generation (RAG) pairs.
- Evaluation: The model was tested against the FLORES-200 Assamese split and a custom 150-prompt multi-domain zero-shot suite.
Key Insights and Ideas
- Efficiency is King: By reducing the "Token Tax," the model can process three times more information in its context window compared to base multilingual models.
- The "Hourglass" Bias: During SFT evaluation, the model showed a bimodal distribution in factuality. It either answered perfectly from memory or triggered a "refusal" template (a side effect of heavy RAG training). This highlights the difficulty of balancing groundedness with open-domain recall.
- Script Invariance: The paper emphasizes the need for Phonetic Transliteration Data Augmentation. Users often type Assamese using Romanized Latin script; models must be trained on dual-script pairs to remain robust.
Key Results
- Tokenization: AssameseLM-E2B required only 35,107 tokens to encode a test corpus that Llama-3.2-3B needed 166,218 tokens for—a nearly 500% reduction in overhead.
- Predictive Power: It achieved a Bits-per-Byte (BPB) score of 0.7488 on FLORES-200, outperforming Qwen2.5-3B (0.9042) and matching the much larger Llama-3.2-3B.
- RAG Performance: The model achieved a 96.7% overall RAG score, with a perfect 100% refusal accuracy on adversarial "unanswerable" prompts, whereas baseline models (Gemma, Llama, Qwen) failed to abstain and instead hallucinated answers.
Limitations and Future Work
- RAG-Prior Refusal: The model occasionally refuses to answer questions it actually knows the answer to if no external context is provided, due to its strict RAG training.
- Parameter Scale: While 2B is efficient, it limits complex reasoning compared to 7B+ models. The authors plan to scale this methodology to larger architectures (7B–9B) in the future.
- Context Window: Future work will focus on expanding the context window to support full-document summarization and multimodal historical manuscript transcription.
Summary updated