Arabic dialect systems are usually scored on clean, controlled test sets. The best known is MADAR's travel corpus, where professional translators rendered the same phrasebook sentences into the dialects of 25 cities. Deployed systems see something else: tweets written in ad hoc phonetic spelling, switching between Modern Standard Arabic and dialect, on topics far from travel. Materials engineering has a long record of this exact mismatch between laboratory and field. Bench tests isolate one stressor, such as heat or one chemical, and then miss what happens when several stressors act together. Stainless steel that resists a pure carbonate melt loses that advantage once a 10% chloride impurity triggers a self-sustaining corrosion cycle (Zeng). Concrete strength measured on small lab cylinders misstates the stiffness of real slabs and columns, because specimen size controls how fast they dry (Hasegawa). That field responded by fitting interaction models across designed multi-factor experiments. Response surface methods, already used to tune multi-factor recipes for alkali-activated binders (Wang et al.), are one example.
This proposal brings that discipline to dialectal NLP. We vary dialect, spelling, domain, code-switching and input length together in a designed grid. We then fit a model of how a system's accuracy depends on each factor and on their pairwise interactions. Finally we test whether that model predicts measured accuracy on real social-media and conversational text better than today's practice of reporting one clean-benchmark score. The point is not another robustness benchmark. The deliverable is a check of whether lab scores forecast field scores, and a breakdown of which combination of stressors opens the gap.
Core Claim
Single-condition dialect benchmarks systematically mispredict field performance, and the error is not additive. A system's drop under unstandardized spelling depends on whether the text is also dialectal and off-domain. Systems that rank first in the lab can therefore rank lower in the field. The literature already shows both halves of this pattern without connecting them. Character n-gram Naive Bayes models win the parallel travel task: SMarT reaches 67.31% macro-F1 and ranks first of 19 systems (SMarT). Neural systems win the Twitter task, where UBC-NLP reaches 71.70% macro-F1 (MADAR shared task). The two tasks differ in many things at once: label granularity, domain, spelling, and sentence-level versus user-level input. Nobody has separated these factors, so nobody can say which shift flips the ranking.
Our hypothesis has three parts. A response-surface model with pairwise interaction terms, fit only on controlled stressed versions of lab data, will predict held-out field accuracy across systems with lower error than a main-effects-only model. It will also order systems more like their field ranking than the raw lab score does. The fitted interactions will name specific dangerous combinations, the language analogue of chloride plus oxide. Our leading candidate is phonetic spelling combined with dialect-specific grammatical markers, because that pairing breaks the tokenizer and the morphological prior at the same time.
Research Program
The base material is MADAR Corpus-26 and Corpus-6. Their parallel structure holds meaning fixed while dialect varies, which makes them the natural analogue of a controlled specimen. On top of this base we define five stressor factors. Each takes a small number of levels, and each level is drawn from a natural rather than invented source wherever possible.
Dialect distance. Levels run from MSA, to a nearby variety, to a distant city variety. This follows the Curras finding that an Egyptian-trained analyzer handles Palestinian text far better than an MSA-trained one: 78.01% versus 63.79% correct analysis (Jarrar et al.).
Spelling. Levels are standardized versus spontaneous spelling. The raw-to-CODA pairs in Curras give us an empirical distribution of how speakers actually misspell relative to the conventional orthography, where 15.9% of tweet words differ from it. We learn character-level rewrite rules from that distribution and apply them to MADAR Levantine sentences. For other regions the noise transfer is admittedly weaker, and we will report that limitation.
Domain. Levels are travel versus other genres. Here we use Curras's own split into social media, TV scripts and folk literature, plus Gulf novels from the Gumar corpus described in the Darwish et al. survey.
Code-switching. Levels are none versus inserted MSA and foreign spans, with insertion rates matched to those that taggers in the AIDA line report in informal text (AIDA).
Input length. Levels are a single sentence versus a concatenated multi-sentence profile. This is the language counterpart of specimen size. Whether longer context dilutes a dialect signal or supplies redundancy that buffers noise is exactly what separates MADAR's two subtasks.
A full crossing of these factors is impossible with natural data, so we use a fractional design: a fraction of the cells is natural, the rest are constructed, and every constructed cell is labeled as such. For each system and task, we fit accuracy as a function of the factor levels with all pairwise interaction terms:
$$\text{acc}(s, x) = \beta_0^s + \sum_i \beta_i^s x_i + \sum_{i<j} \gamma_{ij}^s x_i x_j$$
Here $s$ indexes the system, $x_i$ is the coded level of stressor $i$, $\beta_i^s$ is that stressor's main effect, and $\gamma_{ij}^s$ is the interaction between stressors $i$ and $j$. We then locate each field corpus as a point $x$ in this factor space by measuring its spelling rate, domain and switching rate, and we read off a predicted accuracy.
Tasks and systems. We cover two tasks. The first is dialect identification, for which we collapse city labels to country for comparison against the Twitter task. The second is part-of-speech tagging on Curras. The systems are MNB n-gram models in the style of Eltanbouly et al. and SMarT, MADAMIRA in its MSA and Egyptian versions, fine-tuned mBERT, and an Arabic-centric LLM prompted zero-shot.
Reasoning extension. As a further check, AraDiCE's Egyptian and Levantine versions of MMLU, PIQA and BoolQ (AraDiCE) provide a reasoning task. On that task we can add spelling noise and test the same interaction question for LLMs.
Intensification check. We separately test an analogue of the materials finding that intensified stress changes the failure mechanism instead of merely speeding it up. We inject synthetic character noise at rates from 5% to 50% and extrapolate the resulting accuracy curve down to the natural 15.9% rate. The question is whether the extrapolation matches the drop measured on natural spontaneous spelling. A mismatch would show that common heavy-noise robustness tests do not measure the field mechanism.
Baselines
Raw lab score (do-nothing reference). This is each system's clean MADAR Corpus-26 score, used directly as the prediction of its field score. This is current practice. The strongest reported lab numbers are SMarT's 67.31% macro-F1 and ArbDialectID's 67.32% macro-F1 on Subtask 1 (MADAR shared task). The reported field reference is UBC-NLP's 71.70% macro-F1 on the Twitter subtask. The two numbers are not directly comparable, which is itself the problem we are addressing. We beat this reference if our predictions have lower absolute error against measured field accuracy and higher Spearman rank correlation with the field ordering of systems.
Main-effects-only model (key ablation). This is the same regression with all $\gamma_{ij}$ set to zero. It corresponds to running a battery of separate single-perturbation stress tests and adding up the drops, as in NLI stress testing (Naik et al.) or single-shift dialect evaluation (Romanian–Moldavian domain shift). It uses identical cells, data and fitting procedure, so it isolates whether interactions matter at all. The interaction model must beat it on prediction error for held-out field corpora under leave-one-corpus-out evaluation. Merely fitting the training cells better does not count.
Small field sample. We annotate a matched budget of field sentences from a different region than the target and use them directly to estimate target-field accuracy. This baseline costs the same annotation effort as building the stress grid. For POS tagging, the strongest published transfer number is MADAMIRA Egyptian's 78.01% correct analysis on Palestinian text, and Darwish et al. report 72% POS accuracy on Gulf Arabic. No reported result was found on Lacuna for predicting field accuracy from lab measurements in dialect NLP, so the predictive baselines above have no prior numbers to cite.
Fair-comparison protocol. All systems receive the same training data per task and the same hyperparameter search budget. Neural models get three seeds. The LLM uses fixed prompts.
Why not just…?
Why not just evaluate on a small field sample instead of building a stress grid? This is the most serious alternative. If the target dialect and domain already have field data, a few hundred annotated sentences probably estimate accuracy as well as any model. Our claim is narrower. It concerns forecasting a variety or genre that has no field sample yet, and attributing the gap to specific causes. The cross-region field-sample baseline tests the first part directly. If that baseline matches the interaction model's prediction error, we will report that the predictive claim fails and keep only the diagnostic decomposition.
Why not just normalize spelling to CODA first? If the field gap were mostly spelling, normalization would close it. The fitted model tests this directly: it predicts that normalization closes the gap only if the spelling main effect dominates and its interactions with dialect distance and domain are small. Prior results argue against easy closure. Analyzers fail on 7–18% of Palestinian words (Jarrar et al.), and dialect-specific grammatical markers have no MSA counterpart (Darwish et al.). We will include a normalized pipeline, in the style of dialect-to-standard normalization, as an additional system in the grid.
Why not just use a bigger multilingual model? Scale may shrink every coefficient, but scale alone does not tell us whether lab scores forecast field scores. mBERT beats an SVM on Norwegian dialects, 79.64 versus 77.42 weighted-F1 (Ibrahim). Yet the same paper still fails on intent classes absent from training. Including an LLM and mBERT in the grid shows whether their interaction terms vanish. If they do, that is a useful finding: larger models would make single-condition benchmarks trustworthy.
Proposed Experiments
Experiment 1: prediction. Fit the response-surface model per system on the stressed MADAR and Curras cells. Then predict three held-out field targets: MADAR Twitter country-level identification, Curras raw social-media POS tagging, and Gumar Gulf text. Evaluate with leave-one-corpus-out validation, reporting mean absolute error and Spearman correlation against the raw lab score, the main-effects model, and the field-sample baseline. The decisive check is whether the interaction model predicts the documented ranking flip, where n-gram models lead on travel text and neural models lead on Twitter, from lab cells alone.
Experiment 2: interaction identification. Report the largest $\gamma_{ij}$ terms with bootstrap confidence intervals. We pre-register spelling × dialect distance as the predicted dominant interaction. We also pre-register length × spelling as the term that explains why user-level input favors neural models.
Experiment 3: intensification breakdown. Measure whether accuracy drops extrapolated from heavy synthetic noise match the drop on natural Curras spelling. Compare error types (wrong-dialect confusions versus tokenizer fragmentation) at high and natural noise rates.
Experiment 4: second language family. Repeat the study on the Norwegian NoMusic corpus with dialect, task and unseen-intent factors. This tests whether interaction-driven mispredictions occur outside Arabic.
What would falsify the program. Interaction terms indistinguishable from zero, or no improvement over the raw lab score in predicting field rankings, would mean single-condition benchmarks are adequate. That would be a publishable negative result in its own right.
References
- A Comparative Study of the Corrosion Behavior of Three Stainless Steels in an Eutectic (Li,Na,K) 2 CO 3 Melt with and without (Na,K)Cl Additives at 973K in Air
- INFLUENCE OF DRYING ON THE COMPRESSIVE STRENGTH AND THE MODULUS OF ELASTICITY OF CONCRETE IN STRUCTURES : Part 1 : Relation of Different Curing Environments and Their Effects on the Strength of the Structural Members
- Study on Mechanical Properties and Hydration Characteristics of Bauxite-GGBFS Alkali-Activated Materials, Based on Composite Alkali Activator and Response Surface Method
- The SMarT Classifier for Arabic Fine-Grained Dialect Identification
- The MADAR Shared Task on Arabic Fine-Grained Dialect Identification
- Building a Corpus for Palestinian Arabic: a Preliminary Study
- A panoramic survey of natural language processing in the Arab world
- AIDA: Identifying Code Switching in Informal Arabic Text
- Simple But Not Naïve: Fine-Grained Arabic Dialect Identification Using Only N-Grams
- AraDiCE: Benchmarks for Dialectal and Cultural Capabilities in LLMs
- Stress Test Evaluation for Natural Language Inference
- Dialect Identification under Domain Shift: Experiments with Discriminating Romanian and Moldavian
- Dialect Representation Learning with Neural Dialect-to-Standard Normalization
- CUFE@VarDial 2025 NorSID: Multilingual BERT for Norwegian Dialect Identification and Intent Detection