Dialect identification systems label a whole sentence with one dialect, even though real social-media Arabic rarely has a single dialect. The NADI 2020 analysis found that about half of dialectal tweets carry heavy Modern Standard Arabic patterns, and NADI 2024 annotators judged 70% of tweets valid in two or more dialects (NADI 2024). The signal that separates, say, Egyptian from Levantine often sits in a few affixes or function words, and a sentence-level label averages it away. We propose to treat dialect identification as dense segmentation: every token gets a dialect distribution, trained only from the sentence-level labels that already exist. The sentence labels, multi-label dialect sets, and dialectness scores then follow directly from the token map.
The architectural bet comes from two outside fields. Multimodal learning has shown that when a strong, easy input stream shares an encoder with a weaker one, training leans on the easy stream and the weak one is never learned ("modality laziness") (Sun et al.). In dialect text, the easy stream is the pretrained encoder's topic-level meaning, and the weak stream is spelling and morphology. Lightweight image segmenters handle a related tension by splitting a deep, coarse context branch from a shallow, full-resolution detail branch (ContextNet). We combine the two ideas. A frozen Arabic-dialect encoder supplies context. A tiny character-level branch, trained with its own loss, supplies surface detail. Information flows only one way, from each branch into the fusion head. The same design also lets us ask where the well-documented advantage of language-specific pretraining actually lives.
Core Claim
A dialect tagger built from a frozen language-specific encoder plus an independently supervised character branch of under one million parameters will produce more accurate token-level dialect maps than a fully fine-tuned encoder with a token-classification head. Two conditions matter: the character branch must not receive gradients or activations from the encoder before fusion, and it must carry its own dialect loss. If either condition is removed, the character branch is ignored.
The second claim is diagnostic. Dialect-tweet pretraining clearly helps: MARBERT beat AraBERT and multilingual BERT (mBERT) by 10.7 and 26.5 sarcastic-F1 points on ArSarcasm-v2 (iCompass), and Arabic-centric models beat larger multilingual ones on ArabicMMLU. If much of that gap for dialect identification is a surface and tokenization effect, the isolated character branch should close a large share of it when attached to mBERT or AraBERT. If the gap persists, the advantage lies in contextual knowledge that no surface module can supply. Either result is informative. The cost question is also open: pretraining one encoder per sub-dialect is infeasible, so it matters whether the advantage can be supplied by a small module instead.
Research Program
The model has three parts.
Context branch. This is a pretrained encoder kept frozen apart from small bottleneck adapters, following AlKhamissi et al.. Their work adapted MARBERT to 21-country and 100-province identification and pooled across intermediate layers ("vertical attention") so the classifier could reach lower-layer morphology. We reuse their adapters and layer mixing. We differ from them in adding token-level output and a separate surface channel.
Detail branch. This is a shallow stack of depthwise-separable 1D convolutions over raw characters, in the spirit of ContextNet's detail path and the pointwise decoders of Light-Weight RefineNet. Character outputs are pooled to word positions. Character-level sequence models already segment dialectal Arabic morphology at 91–95% accuracy with little data (Eldesouki et al.). That result supports the claim that surface structure is learnable cheaply. Their work segments morphemes, however, and does not assign dialects.
Fusion. Following LFMIM's one-way flow and per-branch losses (Sun et al.), each branch has its own token-level dialect head and its own sentence-level loss. A late fusion head combines the two. In a variant, each branch also gets a "mimicry" head trained to match the other branch's predictions, borrowed from xMUDA. xMUDA used mimicry to stop camera and LiDAR networks over-relying on one sensor. Our streams are two views of the same text rather than two sensors, so there is no calibration problem, and that removes one of xMUDA's main constraints.
Training without token labels. Token labels do not exist, so training uses multiple-instance aggregation. Each token $t$ in sentence $s$ has a probability $p_t(d)$ of being in dialect $d$, including a Modern Standard Arabic class. Sentence-level predictions are pooled from these:
$$P_s(d) = 1 - \prod_{t \in s} \big(1 - p_t(d)\big), \qquad \widehat{\text{ALDi}}_s = \frac{1}{|s|}\sum_{t \in s} \big(1 - p_t(\text{MSA})\big).$$
$P_s(d)$ is a noisy-OR: a sentence counts as containing dialect $d$ if any token does. $\widehat{\text{ALDi}}_s$ reads the sentence's dialectness score as the fraction of non-standard tokens. The noisy-OR fits the multi-label NADI 2024 labels. For single-label corpora (MADAR, NADI 2020/2021), a softmax over dialects of the pooled log-odds is used instead. The mean-dialectness readout is a hypothesis we test, not an assumption.
Scale of labels. The dense map is trained at a coarse granularity: region groups or the NADI 2024 dialect set plus Standard Arabic. Province labels are evaluated only as a sentence-level readout. Province data are too thin for token supervision; NADI 2020's best province macro-F1 was 6.39%.
Evaluation data. We will build a token-annotated evaluation set of roughly 2,000 NADI 2024 development tweets. Two annotators will mark each token as Standard Arabic, dialect-neutral, or dialect-specific, with the dialect recorded, and we will report inter-annotator agreement. This set is the one real cost of the project. It is necessary because no token-level dialect benchmark appears in the file. We also build controlled probes from MADAR's parallel travel corpus (Bouamor et al.), whose city-parallel sentences allow word-level substitution: one Cairo word is swapped into a Damascus sentence, and the probe checks whether the model localizes it.
Baselines
Naive reference points. A majority-dialect tagger, and a tagger that copies the sentence-level prediction to every token. The copy baseline is the right "do nothing": it measures how much token resolution adds over today's systems.
Strongest existing method (a). A fully fine-tuned MARBERT with vertical-attention layer mixing (AlKhamissi et al.), given a per-token head and trained with the same multiple-instance loss. It is the strongest competitor because it already exposes lower-layer morphology to the classifier, which is the core of our argument. On the sentence side, the reported numbers to beat are 32.26% test macro-F1 on NADI 2021 province-level dialectal Arabic for their ensemble, 50.57 macro-F1 on NADI 2024 multi-label identification (Elyadata), and 0.1403 RMSE on NADI 2024 dialectness (ASOS) (NADI 2024). For token-level dialect segmentation, no reported result was found on Lacuna, so our own fine-tuned tagger is the reference.
Strongest existing method (b). A character n-gram model: a linear SVM, and a CRF over word-window character n-grams, trained from sentence labels by the same aggregation. The file credits such models with matching or beating neural systems across VarDial and MADAR, and names Multinomial Naive Bayes as the winner of the topic-controlled MADAR city task at 67.32% macro-F1.
Fairness. All systems use identical training splits, the same hyperparameter search budget (a fixed number of trials per system), and the same number of seeds. Parameter counts and GPU-hours are reported for each.
Key ablations. - Early fusion: character features concatenated into the encoder's input embeddings. - Joint without isolation: the same character branch trained jointly, with no separate loss and gradients flowing through it. - Remove the context branch: the character branch alone.
The first two ablations isolate the independence ingredient. The third shows what surface features achieve on their own.
What counts as winning. The proposal wins if it beats the fine-tuned tagger on token-level macro-F1 by more than seed variance, while training under a tenth of its parameters, and does not lose on the NADI sentence readouts.
Why not just…?
Why not just fine-tune MARBERT with a token head? This is the real threat. Vertical attention already gives the head access to lower-layer morphology, and full fine-tuning may learn character-like features on its own. The fine-tuned tagger is therefore our primary baseline. We also measure branch reliance directly by zeroing each branch at test time. If the fine-tuned tagger matches us on the annotated set, we will report that the independence claim fails for text, and the pretraining diagnostic stands on its own.
Why not just read token maps off a sentence classifier with attention or gradient attribution? This costs nothing to train, and we include it: integrated gradients and attention rollout over a sentence-level MARBERT, thresholded to tags. Post-hoc attribution explains the classifier's shortcuts, which is the orthographic trap. On topic-matched tweets it should highlight topical words rather than dialect markers. The MADAR substitution probes test exactly this, because the substituted word is known.
Why not just use character n-gram SVMs, which historically won these tasks? They won on small, topic-controlled data such as MADAR travel. Transformer systems dominated noisy Twitter settings: 71.70% for the MADAR Twitter subtask and the top NADI 2020 systems. The SVM is also our detail branch in spirit. The experiment asks whether adding context to it helps, and running the character branch alone answers that directly.
Why not just prompt a large multilingual model? The file's evidence is that multilingual models trail dialect-pretrained ones by wide margins (iCompass, NLEBench+NorGLM). We include one prompted token-tagging baseline on the annotated set so this is measured rather than assumed.
Proposed Experiments
Experiment 1: the dense-segmentation result. On the annotated NADI 2024 set, report token-level macro-F1, and span-boundary F1 for runs of dialect-specific tokens. Compare the full model against every baseline and ablation. If the model is right, early fusion and non-isolated joint training should both sit near the fine-tuned tagger, and only the isolated, separately supervised branch should add a clear gain. The mechanism check is branch reliance under zeroing: in the non-isolated variants, removing the character branch should barely change predictions.
Experiment 2: the readout test. Aggregate token maps into NADI 2024 multi-label predictions and dialectness scores. Aggregate NADI 2021 province predictions as a sentence-level check. Matching 50.57 macro-F1 or 0.1403 RMSE is not required for the thesis. Staying competitive while also producing token maps is required. A dialectness readout far worse than 0.1403 would falsify the mean-dialectness hypothesis.
Experiment 3: the pretraining diagnostic. Swap the frozen context encoder among MARBERT, AraBERT, and mBERT, with and without the character branch, at matched adapter budgets. The quantity of interest is the fraction of the MARBERT–mBERT token-F1 gap that the character branch closes. Plotting that fraction separately for words that the mBERT tokenizer splits into many pieces tests whether the advantage lies in tokenization. This follows the tokenizer-efficiency view of What Language is This? Ask Your Tokenizer.
Experiment 4: robustness. On the MADAR substitution probes, localization accuracy of the single substituted word is the decisive check that the map reflects dialect markers rather than topic. A ContextNet-style gain in accuracy per parameter is expected only if this probe also improves.
References
- NADI 2024: The Fifth Nuanced Arabic Dialect Identification Shared Task
- Layer-wise Fusion with Modality Independence Modeling for Multi-modal Emotion Recognition
- ContextNet: Exploring Context and Detail for Semantic Segmentation in Real-time
- iCompass at Shared Task on Sarcasm and Sentiment Detection in Arabic
- ArabicMMLU: Assessing Massive Multitask Language Understanding in Arabic
- Adapting MARBERT for Improved Arabic Dialect Identification: Submission to the NADI 2021 Shared Task
- Light-Weight RefineNet for Real-Time Semantic Segmentation
- Arabic Multi-Dialect Segmentation: bi-LSTM-CRF vs. SVM
- xMUDA: Cross-Modal Unsupervised Domain Adaptation for 3D Semantic Segmentation
- The MADAR Shared Task on Arabic Fine-Grained Dialect Identification
- NLEBench+NorGLM: A Comprehensive Empirical Analysis and Benchmark Dataset for Generative Language Models in Norwegian
- What Language is This? Ask Your Tokenizer.