Whose Vernacular? Demographic Composition of Dialect Data for Arabic Medical Models

Arabic medical knowledge is written in the formal standard, Modern Standard Arabic (MSA). Patients describe their symptoms in regional dialects, and many write them in Latin-script "Arabizi". Language models are usually taught these dialects by crawling social media, but the people posting on Twitter, YouTube, and Facebook are a skewed slice of Arabic speakers. This proposal asks a concrete question. When a medical model is adapted to dialect using platform text, does it come to understand the dialects of the people who happen to post online and fail everyone else? And is the fix a better sample of real speakers, or no real speakers at all, only synthetic spelling noise added to formal medical text?

We build a dialectal version of an existing Arabic medical exam benchmark. The patient-facing parts of each question are rewritten by native speakers recruited across countries, age bands, genders, and urban and rural origin. We then compare adaptation strategies at matched compute. The central finding we are testing for is a split. Synthetic noise and spelling normalization should close the gap where dialects differ only in spelling. Where dialects use different words for body parts, symptoms, and illnesses, only real vernacular text should help, and it should help only the demographic groups the corpus actually contains.

Core Claim

We claim the dialect gap in domain models has two parts that need different remedies. The first is orthographic: the same word spelled many ways, with up to 27 spellings documented for one Egyptian phrase in A Spelling Correction Corpus for Multiple Arabic Dialects. The second is lexical: a different word for the same concept.

The orthographic part is demographically neutral, and cheap fixes should handle it. The lexical part is inherited from the corpus, so a model adapted on unweighted platform text will show a worst-group failure concentrated in under-sampled speakers. Rebalancing the same number of vernacular tokens by demographic cell should raise worst-group accuracy without lowering the average.

If synthetic noise matches rebalanced real data on lexical items too, the claim is false. That result would itself be useful, because it would tell practitioners they can skip scraping biased platforms altogether.

Why This Is Not Already Done

Three lines of work each cover two of the three pieces, and none has a demographic evaluation.

Dialect pre-training work trains on platform text without auditing who wrote it. AlclaM trains a BERT from scratch on 3.4M YouTube and Facebook comments. AraT5 trains on 1.5B tweets. Both evaluate only general tasks such as sentiment, dialect identification, and generation.

Synthetic-noise work avoids dialect data entirely. Different Perturbations, Different Mechanisms shows that continued pre-training on character- and phoneme-perturbed standard text improves zero-shot dialect performance in 43 of 45 comparisons, by aligning dialect and standard embeddings. ABDUL reaches Algerian and Moroccan dialects from formal corpora through phonemic transliteration. Both evaluate only discriminative tasks. The first paper notes that its gains are smaller for Arabic and that its method is untested where lexical shifts carry distinct meanings. Our lexical split tests exactly that case.

Demographic-bias work shows the harm but not in a diglossic setting. Demographic Inference and Representative Population Estimates from Multilingual Social Media Data finds that raw Twitter samples misestimate census populations with 88% mean absolute percentage error, reduced to 33% by post-stratification (reweighting samples to match known population shares). Towards Understanding the Role of Gender in Deploying Social Media-Based Mental Health Surveillance Models reports a depression classifier with AUC 0.770 on men but 0.702 on women, driven by differing vocabulary. Elucidating Mechanisms of Demographic Bias in LLMs for Healthcare shows that internal demographic representations shift clinical risk predictions. All three are English or dialect-blind. OpinionGPT trains demographic LoRA adapters from Reddit, but it models opinions rather than domain competence, and only in English.

The closest target, MedAraBench, is MSA-only and names that as its limitation. We are the first to measure its dialect gap, broken down by who is speaking.

Research Program

The work has three parts: an evaluation set, a family of adaptation corpora, and a set of controlled adaptation runs.

The evaluation set, MedAraBench-D. We take roughly 1,500 test questions from MedAraBench whose stems contain a patient presentation. Each stem is rewritten by several native writers. Writers come from about eight countries covering Gulf, Levantine, Egyptian, and Maghrebi dialects. Within each country, recruitment is stratified by gender, age (under 30 versus over 50), and urban versus rural origin. Writers are told to express the complaint as they would to a doctor in their own speech and script, which includes Arabizi. Answer options stay in MSA, so any accuracy loss is attributable to reading the vernacular.

Two annotators label every changed span as orthographic-only (the rewrite maps to the same MSA word after CODA normalization, the conventional orthography for dialectal Arabic) or lexical (a different word or construction). This label is what separates the two parts of the gap.

We report a known risk: writer demographics are self-reported rather than inferred. That is deliberate, because automatic demographic inference is unreliable for Arabic names, as Inferring Gender from Names on the Web reports for non-Western naming conventions.

The adaptation corpora. We assemble a platform corpus of Arabic health-related posts and comments from the same platform types used by AlcLaM and AraT5. We then audit it at the account level. Region comes from a country-level dialect identifier trained on MADAR Twitter data (The MADAR Shared Task). Gender and age come from manual annotation of a sample of a few thousand accounts, which is used to calibrate a profile classifier. From the audited corpus we draw three token-matched subsets: an unweighted sample, a sample post-stratified to population shares by country, gender, and age, and single-cell samples for dose-response experiments. The post-stratification follows the multilevel-regression-and-post-stratification (MRP) logic of Wang et al.

The adaptation runs. All runs start from the same open model, Llama-3.1-8B, which MedAraBench reports at 17% accuracy base and 32% after QLoRA fine-tuning on its training split. Each run applies LoRA continued pre-training at a fixed token budget, then the identical MedAraBench QLoRA fine-tuning step. We replicate the key contrast on Fanar to check that results are not specific to one base model.

Baselines

The naive reference point is the MSA-only pipeline: Llama-3.1-8B with QLoRA on the MedAraBench training split and no dialect exposure. MedAraBench reports 32% accuracy for this setup on the original MSA test. We re-run it and score it on both the MSA and dialect versions. Its drop from MSA to dialect defines the gap every other method must close. Beating it means closing a statistically significant share of that drop, per cell, with paired bootstrap tests over questions.

The strongest cheap competitor is normalize-then-answer. Dialect input is passed through a CODA-style spelling normalizer (an MLE normalizer reaches 92% accuracy from 125 sentences in the CODA corpus paper) or through dialect-to-MSA translation in the style of Sajjad et al.. The MSA-only model then answers. The normalizer and translator get training data from the same annotation budget as our methods. We expect this baseline to win on orthographic items and lose on lexical ones.

The strongest existing method is synthetic-noise continued pre-training: CHAR and PHON perturbations from Srivastava & Chiang, applied to Arabic medical text at our exact token budget and LoRA rank. They report an average dialect gain of 3.0 points on general tasks. There is no reported result on Lacuna for synthetic-noise pre-training on any medical or dialectal medical benchmark, so we establish this number ourselves. It is the baseline the proposal must beat on lexical items in under-sampled cells.

The unweighted platform-corpus run, the AlcLaM and AraT5 recipe transplanted to health text, is the ablation that isolates our key ingredient. It uses the same tokens, the same model, and the same compute as the post-stratified run, and differs only in demographic composition. Post-stratification counts as a win only if it raises worst-cell accuracy by a significant margin while keeping mean dialect accuracy within one point.

As a ceiling reference we score GPT-o3, which MedAraBench reports at 76.5% on the MSA test, and Fanar, reported at 49.8%, zero-shot on MedAraBench-D. This tells us whether scale alone erases the demographic gradient.

Why not just…?

Why not just use a frontier model? MedAraBench shows proprietary reasoning models far ahead on MSA: 76.5% for GPT-o3 against 54.7% for the best open model. Scale is therefore a real competitor on average accuracy. Our question, though, is whether the gap is uneven across speakers. The frontier model's pretraining data comes from the same skewed web. If GPT-o3 shows a flat dialect gradient across demographic cells, the corpus-composition story is weakened for large models, and we will report that. The open-model results would still matter for deployable, locally hosted systems.

Why not just normalize or translate dialect into MSA first? For spelling variation this probably works, and our orthographic split is designed to let it win there. For lexical substitution, normalization has nothing to map to. The halal/haram benchmark study (Do LLMs Distinguish Between Halal and Haram?) documents translation pipelines losing domain nuance, and MedAraBench warns that machine translation drops clinical detail. The lexical-item accuracy of this baseline, per cell, settles the question directly.

Why not just add synthetic noise to clean medical text and never touch biased platforms? This is the most serious threat, because it is demographically neutral by construction and it already works on general dialect tasks. Its mechanism, embedding alignment between perturbed and clean forms, predicts success on orthographic items and no benefit on lexical ones. That is a falsifiable prediction. If noise pre-training matches post-stratified real data on lexical items in rural and older cells, we will conclude real vernacular data is unnecessary for this purpose. We will also test the combination: noise pre-training followed by a small rebalanced real-data stage.

Why not just fine-tune on dialect examples? No dialectal medical training set exists. DIALECTBENCH also reports a fine-tuning paradox, where fine-tuning on small, noisy variety data widened disparities relative to zero-shot transfer. We include a run that fine-tunes on a small held-out slice of MedAraBench-D rewrites and check whether it reproduces that widening.

Proposed Experiments and Decisive Checks

The primary metrics are per-cell accuracy on MedAraBench-D, worst-cell accuracy, and the MSA-to-dialect drop, each split into orthographic and lexical items. We also track MSA accuracy to detect forgetting.

The first decisive check is the interaction between method and item type. Normalization and noise pre-training should close most of the orthographic gap but leave the lexical gap. Real-data pre-training should reduce both.

The second is the composition test. With tokens held equal, post-stratified data should beat unweighted data on worst-cell lexical accuracy, and the gain should land in the cells that were under-represented before reweighting.

The third is a dose-response curve. For three cells, we vary each cell's share of the pre-training corpus from 0% to 30% and plot that cell's lexical accuracy. A monotone slope, together with flat curves for the other cells, would show that the model learns vernacular medical vocabulary from specific speakers rather than from dialect exposure in general.

The fourth is an internal check. Borrowing the probing approach of Ahsan et al., we test whether unweighted pre-training makes dialect region linearly decodable in intermediate layers more strongly than rebalanced pre-training does, and whether that decodability predicts per-cell error.

Any of these checks failing narrows the claim. The first failing would mean the orthographic/lexical split does not explain the gap. The second failing would mean corpus composition does not matter at this scale. Either way, the benchmark and the per-cell measurements remain a lasting contribution.