Anchoring Informal Word Forms to Formal Ones in Small Local Language Models

Small language models pre-trained on a single language or region often beat much larger multilingual models on that language. The corpora those local models learn from are mostly news, Wikipedia and Bible translations, though, while the text people actually want analyzed is tweets, forum posts and dialect. On that informal text, words appear with dropped diacritics, dialectal spellings, stretched letters and new affixes. Each variant is rare or absent in the pre-training data, and the tokenizer breaks it into meaningless fragments. This proposal starts from a simple hypothesis. Much of the local-model advantage, and much of its erosion under domain shift, comes from how often the test-time word forms were seen during pre-training. If that is true, the cheapest fix is to teach the model that a rare informal form means roughly what its frequent formal counterpart means, and to do so before spending scarce informal data on further training.

Concretely, we propose to add frequent informal variants to a local model's vocabulary. Each new variant's input vector is initialized as a weighted mix of the vectors of formal words it is likely a variant of, found through spelling rules and context similarity. A short round of continued training follows on whatever small informal corpus exists. We will test whether this beats standard continued pre-training on informal text specifically when that corpus is small, and whether the gains fall on the test items whose words were rare in pre-training. That second check is what would make the mechanism claim, and not just a leaderboard number, believable.

Core Claim

The claim has two parts, and both can be falsified. The first is diagnostic. On informal benchmarks, the accuracy gap between a formal-data local model and a model pre-trained on informal text should concentrate in test items whose words are rare in the formal pre-training corpus or split into many subword pieces. The gap should not be spread evenly. The second is interventional. Anchoring informal variants to formal counterparts at initialization should recover a large share of that gap using unlabeled informal corpora orders of magnitude smaller than those used by domain-adaptive pre-training. The advantage should shrink as the informal corpus grows, so the curves should cross. If plain continued pre-training matches the method at every corpus size, the anchoring idea is wrong and we will say so.

The motivating evidence comes from three places. iCompass reports that MARBERT, pre-trained on dialectal Arabic tweets, reaches 57.7 F1 on sarcastic tweets versus 47.0 for the Modern Standard Arabic model AraBERT and 31.2 for multilingual BERT. So being both language-specific and domain-matched matters, and language-specific alone is not enough. AfriSenti finds that Twitter-trained XLM-T beats larger general models, and that the Africa-centric AfriBERTa fails on Moroccan and Algerian Arabic. TWEETQA shows that only about 10% of its vocabulary is shared with a Wikipedia QA set. None of these papers asks which test items account for the gap, or whether the gap can be closed without a large informal corpus. Collecting such a corpus for low-resource languages is hard because language identification breaks on noisy web text, as CommonLID documents.

Relation to Prior Work

Three existing lines of work supply pieces of this idea and also limit what it can be. Classical sparsity work shows that rare items should borrow statistical support from related ones. Tied-Mixture Language Modeling in Continuous Space shares continuous mixture components across the vocabulary so that rare words lean on neighbors, and our initialization applies the same intuition to the input vectors of a pre-trained Transformer. Factored models give a more explicit version. Factored Language Model based on Recurrent Neural Network feeds lemma, stem and part-of-speech streams into the network. Arabic Language Modeling with Finite State Transducers raises test 4-gram coverage from 0.37% to 9.62% by decomposing words with hand-written morphological grammars. Both depend on parsers or grammars that break on unstandardized informal text. That is a constraint on design, so we deliberately use no parser, only spelling rules and distributional evidence. The Web as a Baseline supports the premise that more raw data alone does not remove the long tail of rare forms.

The language-specific work supplies the base models and the vocabulary-expansion technique. SeaLLMs merges new tokens into LLaMA-2's vocabulary and cuts Thai from 4.29 to 1.57 tokens per word. Its added tokens come from a formal translation tokenizer, and it does not evaluate on informal text. AfroLM shows that careful use of a small (~0.73 GB) news corpus can beat much larger models. IndoNLG shows that 132M-parameter local models compete with mBART-Large, though only on mostly formal generation tasks. Our work adds an informal-domain step that these papers leave out: expanding the vocabulary with dialectal and informal variants and initializing the new vectors from formal anchors without parallel formal–informal text.

The domain-adaptation work supplies the strongest competitor. fBERT continues masked-language-model training on 1.4M social media comments. The winning AfriSenti shared-task system described in AfriSenti-SemEval uses language-adaptive and task-adaptive continued pre-training. These methods work when informal text is plentiful. Our claim is limited to the regime where it is not.

Research Program

The first component is a sparsity profile for each benchmark–model pair. For every test example we compute two quantities over its words: their frequency in the model's pre-training corpus, or a matched proxy corpus when the original is unavailable, and the tokenizer's fertility, meaning subword pieces per word. We then bin examples by these quantities and report accuracy per bin for formal-data local models, informal-data local models and multilingual models. This tells us whether the local-model advantage and its loss under domain shift really track sparsity.

The second component is variant-anchored vocabulary expansion. From a small unlabeled informal corpus we select the most frequent word forms that the base tokenizer splits heavily, and add them as new tokens. For each new form $v$ we build a candidate set of formal words $f_1, \dots, f_k$ drawn from the base vocabulary and frequent formal words. Candidates come from cheap, language-appropriate spelling rules, such as removing diacritics for Yorùbá, collapsing repeated letters, and small edit distance after vowel dropping. We then initialize

$$e(v) = \sum_{j=1}^{k} w_j \, e(f_j), \qquad w_j \propto \exp\big(\mathrm{sim}(c_v, c_{f_j}) / \tau\big),$$

where $e(\cdot)$ is an input vector, $c_v$ is the base model's average contextual representation of $v$'s occurrences in the informal corpus, $c_{f_j}$ is the same quantity for $f_j$ computed on formal text, $\mathrm{sim}$ is cosine similarity and $\tau$ is a temperature. Forms with no confident candidate fall back to the mean of their subword pieces. A short continued masked-language-model run on the same small informal corpus follows, and then task fine-tuning.

The third component is a corpus-size sweep. Every adaptation method is run with unlabeled informal corpora from roughly $10^4$ to $10^6$ tokens and, where available, beyond. This maps where anchoring helps and where ordinary continued pre-training catches up.

Baselines

The naive reference point is the formal-data local model fine-tuned directly on the labeled task with no adaptation. For African languages this is AfroXLMR-large, which AfriSenti reports as its top baseline at 71.2 average macro F1. For Arabic it is AraBERT, which iCompass reports at 47.0 F1 on sarcastic tweets and 68.7 F-PN (a sentiment F1 averaged over the positive and negative classes) on ArSarcasm-v2 development data. A classical Naive Bayes classifier is also included, because the Minangkabau study found it beat zero-shot multilingual BERT by nearly 6 F1. Any method that loses to it is not worth pursuing.

The strongest existing method is language-adaptive plus task-adaptive continued pre-training, the recipe of the AfriSenti shared-task winner. AfriSenti-SemEval reports 71.31 weighted F1 on the monolingual track for that system. It is the right competitor because it attacks domain mismatch directly using the same unlabeled data we use. We will run it on the identical informal corpus at each size in the sweep, with the same number of update steps and the same hyperparameter search budget as our method. This way the only difference is the vocabulary expansion and its initialization. Beating it means higher macro F1 averaged over at least three seeds, with non-overlapping confidence intervals, at the small end of the sweep. We do not expect to beat it at the large end.

The upper reference for Arabic is MARBERT itself, pre-trained from the start on dialectal tweets, at 57.7 F1 on sarcastic tweets and 72.95 F-PN on development data per iCompass. A strong result would be anchored AraBERT, adapted with a small dialect corpus, recovering a substantial share of the AraBERT-to-MARBERT gap.

Two ablations isolate the key ingredient. The first is vocabulary expansion with the new vectors initialized as the mean of their old subword pieces, which is the common default for vocabulary extension. The second is expansion with random initialization. If anchored initialization does not beat subword-mean initialization, the formal-anchor idea adds nothing beyond vocabulary expansion itself. A third control is subword regularization, which randomly re-segments words during training so the model sees many splits. It tests whether plain robustness to fragmentation is enough without any anchoring.

Why not just…?

Why not just run continued pre-training on the informal text? This is the real threat. fBERT and the AfriSenti winner show that it works. Its weakness is data: for languages like Xitsonga, with about 1.2k labeled tweets in AfriSenti, the matching unlabeled informal text is also scarce and hard to identify reliably. The corpus-size sweep settles the question. If ordinary continued pre-training matches anchoring even at $10^4$–$10^5$ tokens, we drop the method claim and keep only the diagnostic result.

Why not just normalize informal text into formal text first? A pipeline that restores Yorùbá diacritics, using the system from Improving Yorùbá Diacritic Restoration (20.40% word error rate for the best model on its evaluation), and then applies the formal model is a strong and simple competitor. It has two costs. Normalization errors propagate into the classifier, and normalization may erase signals such as stretched letters that carry sarcasm and sentiment. We will run this pipeline on the Yorùbá AfriSenti split and compare the two, with results broken down by sparsity bin.

Why not just prompt a large multilingual model? The file suggests scale does not remove the language gap. SeaLLM-7B beats ChatGPT-3.5 on translated math reasoning in Thai, 62.0 versus 35.8, per SeaLLMs. GPT-4 falls from about 80% on translated MMLU to 72.5% on native ArabicMMLU, per ArabicMMLU. We found no reported result on Lacuna for frontier-model prompting on AfriSenti or ArSarcasm-v2. We will therefore include one prompted large model as a reference and analyze it with the same sparsity bins, without treating it as a like-for-like competitor on cost.

Proposed Experiments

The main benchmarks are the AfriSenti monolingual tracks, focusing on Yorùbá, which has heavy diacritic variation; Hausa, which is the largest; Nigerian Pidgin; Moroccan and Algerian Arabic, where Africa-centric models are known to fail; and Xitsonga, which is the smallest. ArSarcasm-v2 is the dialect-sarcasm benchmark, using the 80:20 split from iCompass. The metrics are macro F1 for AfriSenti, and F1 on sarcastic tweets plus F-PN for ArSarcasm-v2. Because the sarcastic class is severely imbalanced (2,168 of 12,548 tweets), all results are reported per class. The Minangkabau sentiment set serves as a secondary check where Indonesian is the formal anchor language. It was translated rather than collected natively, so it tests the anchoring mechanism more than realistic informality.

Three checks decide the outcome. First, the diagnostic check: in the sparsity profile, the gap between formal-data and informal-data models should be at least twice as large in the highest-fertility, lowest-frequency tercile as in the lowest. If the gap is flat across bins, the sparsity explanation fails. Second, the intervention check: at small corpus sizes, anchored expansion should beat both continued pre-training and subword-mean expansion, and its gains should fall mainly in the sparse tercile. Gains spread evenly would suggest a generic regularization effect rather than anchoring. Third, the crossover check: as the informal corpus grows, anchored expansion's advantage over continued pre-training should shrink toward zero. That pattern is what we would expect if the method substitutes for missing informal data, and the size at which the curves cross tells practitioners when anchoring is no longer worth the effort.