When a large language model is asked to write in a regional dialect such as Lebanese Arabic, Egyptian Arabic, or Bavarian, it often produces something that looks dialectal without being dialectal. It sprinkles in a few high-frequency signature words, and the grammar, idiom and cultural content underneath stay those of the standard language, or of English. Art historians studying cultural fusion have a name for the visual version of this: pasting traditional motifs onto a foreign frame. The China-chic literature argues that it produces "aesthetic fatigue," and that real synthesis has to happen at the structural layer (China-chic: From Chinese Elements to International Trend). We propose to make this distinction measurable for dialect generation. We would separate the surface markers a model emits from the structural dialect grammar and culturally grounded content it actually realizes. We would then use the gap between the two to decide, case by case, whether a model should write in full dialect, patch its weak layers with external tools, or openly fall back to a standard or hybrid register.
The decision step borrows its logic from SWOT analysis, the strategy-planning method that crosses an entity's internal strengths and weaknesses with external opportunities and threats. It does not borrow SWOT as a prompt template. Here "internal" means a measured profile of what a given model can and cannot do in a given dialect, layer by layer. "External" means the resources available in the deployment setting and the cost of getting it wrong there. The strategy is chosen by crossing the two, and the measurement is what makes the choice non-arbitrary. The two-sentence version is this. Models fake dialects by overusing surface words, and the usual automatic checks reward the fake. We build a layer-by-layer test that catches the gap and use it to route each generation request to the strategy the model can actually support.
Core Claim
Our first claim is empirical. In generation into dialect, LLM outputs show a systematic surplus of surface dialect markers relative to human references and a deficit of structural dialect features. Surface markers are signature lexical items and phonetic spellings. Structural features include morphosyntax such as the Palestinian negation circumclitic documented in the Curras corpus (Nizar Habash) and dialect-specific aspect and future markers. The same outputs also show a further deficit in culturally grounded content.
Our second claim is that the evaluators people reach for first are biased toward pasted text. Character n-gram dialect identifiers and BLEU fall into this group, and possibly also learned metrics such as xCOMET, a neural translation-quality scorer that predicts human ratings. Character n-gram dialect identifiers work very well for classification (Sohaila Eltanbouly et al.; Karima Meftouh et al.), but they key on exactly the surface signals that pasting inflates.
Our third claim is practical. A routing policy derived from the per-layer audit gives outputs that native speakers judge more authentic and more acceptable than always generating in dialect or always using the standard. Its biggest gains should come where the model is weakest.
The idea connects three bodies of work that do not currently talk to each other. Sociocultural critique of cultural NLP warns that models learn "higher-order indexicality," meaning stereotyped talk about a culture rather than situated practice (Culture is Not Trivia: Sociocultural Theory for Cultural NLP). Dialect pasting is the linguistic form of that failure, and we make it countable. XCR-Bench shows that models degrade from visible to semi-visible to invisible cultural layers (XCR-Bench: A Multi-Task Benchmark for Evaluating Cultural Reasoning in LLMs), but it scores sentence-level substitution of culture-specific items in broadly national settings, not grammar inside dialectal text. BavGround finds a "tourist versus local" gap, with open-weight models at 63.1% on general questions against 46.7% on source-grounded ones and falling to 45.9% when prompted in Bavarian (BavGround: A Benchmark for Regional Cultural Grounding and Dialect Competence in Bavarian), but it is multiple-choice only. We carry both layered analyses into open generation and separate the layers inside a single output.
What Gets Measured
For each output we compute three quantities against a native human reference for the same source content.
The first is surface marker density $s$. It is the rate of dialect-signature lexical items and dialectal spellings per token, normalized by the same rate in the human reference. A value of 1 means the output matches native density, and values above 1 mean over-marking.
The second is structural realization $r$. We identify the contexts in the source that a native speaker would render with dialect-specific morphosyntax, such as negation, future and progressive marking, and pronominal clitics. We then compute the fraction of those contexts in which the output uses the dialect form. Structural features for Arabic are detected with a multi-dialect morphological analyzer in the MAGEAD tradition, which separates roots from regional patterns using finite-state machines (Nizar Habash et al.). Its detections are validated against a hand-annotated sample.
The third is grounded-content fidelity $g$. It is the fraction of culture-specific items in the reference that are preserved or appropriately localized rather than replaced by standard-language or Western defaults. It is annotated following the visible, semi-visible and invisible layering used by XCR-Bench.
The pasting gap is then
$$\Delta = \min(s, 2) - r,$$
where capping $s$ at 2 keeps a few extreme outputs from dominating. A large positive $\Delta$ means the text looks more dialectal on the surface than its grammar supports. We report $g$ separately rather than folding it in, because content failures and grammar failures call for different fixes.
Research Program
The work has three stages. The first stage builds the evaluation set.
For Lebanese Arabic we reverse the Language Wave dataset, which contains 2,947 authentic podcast sentences with professional English translations (Can LLMs Translate Cultural Nuance in Dialects? A Case Study on Lebanese Arabic). The English becomes the source and the native Lebanese becomes the reference. This direction is important because the authentic dialect side is human-produced, unlike corpora translated from Modern Standard Arabic (MSA), which that paper shows lack real colloquial nuance.
For Egyptian Arabic we use the MSA children's stories of the AraSim parallel corpus (AraSim: Optimizing Arabic Dialect Translation in Children's Literature with LLMs and Similarity Scores). Its dialect side was produced by Claude and AraT5, so it is not a gold reference. We therefore have native Cairene speakers rewrite a subset of stories at full-story level, which also addresses the narrative fragmentation that AraSim attributes to line-by-line translation.
For Palestinian Arabic we build a smaller probe set of sentences whose standard forms require the negation circumclitic and other Curras-attested structures.
For Bavarian, which lacks morphological tooling, we measure only $s$ and $g$. Native writers render BavGround question stems and source passages into dialect. This gives a transfer test of the method where the structural layer must be hand-annotated.
The second stage is the audit. For each model and dialect we estimate $s$, $r$ and $g$ on a held-out calibration split. The result is the internal side of the analysis: a per-layer profile of strengths, where the model reaches native levels, and weaknesses, where it falls short.
The external side is specified per deployment setting. It records which opportunities exist, such as a morphological analyzer usable for post-editing or a local corpus for retrieving grounded content. It also records which threats apply, such as an audience of children, where stereotyped dialect is costly, or a mixed audience, where full dialect limits reach. The children's-story setting and the podcast-dialogue setting have deliberately different threat profiles.
The third stage is routing. The cross-references of the SWOT matrix map onto four concrete generation strategies.
The strength-opportunity strategy is unconstrained full-dialect generation. It is used where $r$ and $g$ are near reference levels.
The weakness-opportunity strategy uses external resources to repair what the model lacks. This parallels how the regional-governance literature describes zones that close internal gaps through outside partnerships (Research on Eco-environment Collaborative Governance of Wanjiang Urban Belt under the Perspective of Yangtze River Delta Regional Integration). Concretely, a weak structural layer triggers morphology-guided post-editing, and a weak content layer triggers retrieval of grounded passages.
The strength-threat strategy is dialect generation with marker density explicitly constrained toward the reference rate.
The weakness-threat strategy is an honest fallback to the standard language, or a declared light-dialect register, when neither the model nor the available tools can support authentic output.
The router uses only quantities measured before generation. These are the audit profile plus a per-input estimate of how many structure-demanding contexts and culture-specific items the source contains. The rules are fixed on the calibration split and then frozen.
The lesson from Project Loon applies directly. That project failed because internal engineering competence was assessed without regard to external constraints (Lessons From the Commercial Failure of Project Loon for 6G Research Roadmap Design), and dialect deployment fails the same way when fluency is judged without regard to audience. Kwon et al. found that LLM negotiators state priorities they do not act on (Evaluating Behavioral Alignment in Conflict Dialogue: A Multi-Dimensional Comparison of LLM Agents and Humans). We expect a parallel here, namely that models asked for a dialect will claim to produce it while realizing it only on the surface. This is why the routing relies on measured profiles rather than on the model's self-report.
Baselines
The naive reference points are always-dialect zero-shot generation and always-standard generation. For always-standard we use MSA for the Arabic sets and Standard German for the Bavarian set. Both are run on the same models and inputs. Always-dialect is what practitioners do by default. Always-standard is the floor that routing must beat on authenticity without losing on acceptability.
The strongest existing generation systems are Arabic-adapted large models. Jais-adapted-70B and Command-R+ are the best LLMs in the Lebanese study, which reports xCOMET "in the 0.75–0.8 range" on Language Wave for Lebanese-to-English translation, against "around 0.65" for NLLB-200 and Google Translate (Can LLMs Translate Cultural Nuance in Dialects? A Case Study on Lebanese Arabic). These numbers are for the opposite translation direction. No reported result was found on Lacuna for English-to-Lebanese generation on Language Wave, so we will establish that number ourselves.
For Egyptian, the reference system is AraSim's fine-tuned AraT5, which reports semantic similarity of 0.97 and BLEU of 0.087 (AraSim). Both scores are computed against machine-produced references, so we re-score that system against our native rewrites. We also include NLLB-200 as the standard translation baseline.
Every system gets the same inputs, the same decoding budget, and the same total number of model calls, with tool calls counted. Any prompt tuning is done on the calibration split only.
Three ablations isolate the key ingredient, which is routing on measured internal weaknesses. The first gives a generic plan-then-generate prompt of matched length and call count, with no audit. The second applies all four strategies uniformly, for example always post-editing. The third replaces the measured audit with the model's own self-assessment of its dialect competence.
Beating the baselines means two things. Routing must reduce $\Delta$ and raise native-speaker authenticity ratings relative to always-dialect, with non-overlapping bootstrap intervals. Routing must also not lose on acceptability to always-standard in the high-threat children's setting. If the matched generic-plan ablation does as well as routing, the SWOT-derived structure has contributed nothing beyond extra computation, and we will say so.
Why not just…?
Why not just use a dialect identifier as a quality gate? Character n-gram and BERT-based identifiers are strong and cheap (Sohaila Eltanbouly et al.; Multi-dialect Arabic BERT for Country-level Dialect Identification). We predict that they reward pasting, because over-marked text is more confidently "dialectal" than native text. We test this directly. We correlate identifier confidence with $s$, $r$ and native authenticity ratings, and we build a controlled set in which we inject signature words into standard-grammar sentences. If identifier confidence tracks $r$ and human ratings as well as our measure does, a cheap gate suffices and our measurement layer is unnecessary.
Why not just score outputs with xCOMET? It is the best-validated metric available, correlating at $r = 0.606$ with human judgment against $r = 0.098$ for BLEU on dialect translation (Can LLMs Translate Cultural Nuance in Dialects?). That validation was in the dialect-to-English direction, however, where meaning preservation dominates, and nothing in the file tests whether xCOMET penalizes standard grammar dressed in dialect words. This is a real threat. We will include xCOMET and a dialect-aware embedding similarity computed with Swan-Large, which scores 70.45 on ArabicMTEB's dialect tasks (Swan and ArabicMTEB), as competing evaluators on the injected-marker set. If either one already separates pasted from native text, we will drop our structural detector in its favor and keep only the routing contribution.
Why not just prompt the model with a dialect-speaker persona or a careful plan? Persona prompting reproduces stereotypes (Culture is Not Trivia). Leading models still fall to 49.17% on Indian conversational norms in the Culturally-Aware Conversations benchmark (Culturally-Aware Conversations: A Framework & Benchmark for LLMs). We expect a persona to raise $s$ without raising $r$. The plan-prompt variant is the matched ablation described above, so this alternative is settled by experiment rather than by argument.
Why not just fine-tune on more dialect data? Fine-tuning is the right fix where data exists, and AraSim's fine-tuning gave modest gains (BLEU 0.082 to 0.087). Fine-tuning and routing are not competitors, though. We run the audit on a fine-tuned model as well. If fine-tuning closes $\Delta$ everywhere, the router should learn to send almost everything to the full-dialect strategy, and that outcome is itself a useful finding about when routing stops mattering.
Decisive Checks
The program depends on three checks.
The first is whether pasting exists at scale. Across models and the three Arabic dialects, the median $\Delta$ for always-dialect output must be reliably above zero, and it must be largest on the Palestinian structural probes and the BavGround-grounded content. If outputs match native structure, the premise fails.
The second is whether $\Delta$ predicts what natives care about. Native-speaker authenticity ratings, collected blind with explicit instructions to judge grammar and cultural fit and not only meaning, must correlate with $\Delta$ beyond what xCOMET and identifier confidence explain. The Lebanese study notes that its own annotators were not asked to judge this.
The third is whether measured routing beats its unmeasured twins. It must outperform both the generic-plan ablation and the self-assessment ablation on authenticity at equal compute, with the largest margins in the children's-story setting, where the threat of stereotyped output is highest.
If all three hold, the result is a reusable test for a failure that current dialect benchmarks cannot see, plus a principled rule for when a model should not speak in dialect at all.
References
- China-chic: From Chinese Elements to International Trend
- Building a Corpus for Palestinian Arabic: a Preliminary Study
- Simple But Not Naïve: Fine-Grained Arabic Dialect Identification Using Only N-Grams
- The SMarT Classifier for Arabic Fine-Grained Dialect Identification
- Culture is Not Trivia: Sociocultural Theory for Cultural NLP
- XCR-Bench: A Multi-Task Benchmark for Evaluating Cultural Reasoning in LLMs
- BavGround: A Benchmark for Regional Cultural Grounding and Dialect Competence in Bavarian
- MAGEAD: A Morphological Analyzer and Generator for the Arabic Dialects
- Can LLMs Translate Cultural Nuance in Dialects? A Case Study on Lebanese Arabic
- AraSim: Optimizing Arabic Dialect Translation in Children’s Literature with LLMs and Similarity Scores
- Research on Eco-environment Collaborative Governance of Wanjiang Urban Belt under the Perspective of Yangtze River Delta Regional Integration
- Lessons From the Commercial Failure of Project Loon for 6G Research Roadmap Design
- Evaluating Behavioral Alignment in Conflict Dialogue: A Multi-Dimensional Comparison of LLM Agents and Humans
- Multi-dialect Arabic BERT for Country-level Dialect Identification
- Swan and ArabicMTEB: Dialect-Aware, Arabic-Centric, Cross-Lingual, and Cross-Cultural Embedding Models and Benchmarks
- Culturally-Aware Conversations: A Framework & Benchmark for LLMs