Is the Dialect Gap a Memorization Gap?

Large language models answer reasoning questions noticeably worse when the same question is written in African American English (AAE) rather than Standard American English (SAE). On MMLU, GPT-4 loses 10.7% accuracy, and parallel reasoning queries show average relative drops above 10%, reaching 30% on planning. These numbers are usually read as evidence that models reason poorly over non-standard grammar. There is a confound nobody has removed. The standard-English versions of these benchmarks circulate verbatim on the public web and have very likely been memorized during pretraining, while the dialect versions are new strings the model has never seen. Part of the "dialect gap" may therefore be a "memorization gap": the model recognizes the SAE item but not its AAE rewrite.

This note proposes to measure how much of the published dialect gap comes from each source. The method rests on a single quantity, the transfer ratio: when a model has memorized a benchmark item in SAE, what fraction of that memorization advantage survives when the item is rewritten in AAE? Prior work points in two directions on this. Contamination is known to cross whole language barriers, which suggests memorization should survive a dialect rewrite easily. The dialect-gap literature attributes the drop to missing dialect exposure, which suggests it does not. Measuring the transfer ratio directly, through controlled contamination and contamination-free item generation, settles which reading is right and corrects the fairness numbers the field currently reports.

Core Claim

Write the observed accuracy gap between SAE and AAE versions of a benchmark as

$$ \Delta_{\text{obs}} = \Delta_{\text{true}} + (1-\tau)\, G_{\text{SAE}}, $$

where $\Delta_{\text{true}}$ is the gap the model would show on items it has never seen in either dialect, $G_{\text{SAE}}$ is the accuracy gain that contamination gives on SAE items, and $\tau$ is the transfer ratio, the gain on AAE versions of contaminated items divided by $G_{\text{SAE}}$. If $\tau \approx 1$, contamination inflates both dialects equally and the published gaps reflect genuine dialect brittleness. If $\tau$ is well below 1 and $G_{\text{SAE}}$ is large, as contamination estimates of 1% to 45% on popular QA benchmarks suggest it can be (Benchmark Data Contamination of Large Language Models: A Survey), then a measurable share of the dialect gap is an evaluation artifact. Static dialect audits would then overstate reasoning bias and understate how much models simply recall.

The competing predictions come from two bodies of work. Data Contamination Can Cross Language Barriers fine-tuned LLaMA3-8B and Qwen1.5-7B on French, Chinese, and other translations of MMLU, ARC, and MathQA, and found that this inflated English scores without leaving surface traces. LLaMA3-8B MathQA rose from 42.01% to over 95%. If memorization survives a full language change, a dialect rewrite that keeps most content words should transfer it almost completely, which predicts $\tau$ near 1. In contrast, Assessing Dialect Fairness and Robustness of Large Language Models in Reasoning Tasks (ReDial) and Disparities in LLM Reasoning Accuracy and Explanations attribute their drops to limited dialect exposure and never check whether the SAE source items were memorized. Yao et al. tested only standardized national languages, so the dialect case is genuinely open. The asymmetry also matters in its own right: memorization might transfer from French to English but not from SAE to AAE, because AAE morphosyntax such as negative inversion and habitual "be" breaks the specific token sequences a model has memorized.

Research Program

The program has three parts. Each estimates a different term in the decomposition, and the parts check one another.

The first part is a controlled contamination dose-response experiment, which estimates $\tau$ directly. Following Yao et al.'s protocol, we fine-tune LLaMA3-8B and Qwen1.5-7B on SAE versions of a random half of MMLU and MathQA items at several exposure levels: none, one epoch, and multiple epochs. We also train in the reverse direction, on AAE versions of the items. The held-out half serves as clean control items. AAE versions come from the 11-rule morphosyntactic pipeline validated by native speakers in the Disparities paper. Because we choose which items are contaminated, the gain on contaminated items in each dialect is measured exactly, and $\tau$ follows from the ratio of those two gains. The reverse arm tests a second question: whether AAE exposure contaminates SAE evaluations, and therefore whether transfer is symmetric.

The second part is an item-level audit of models as they ship, without any injected contamination. For off-the-shelf open models, we estimate each benchmark item's prior exposure in three ways: verbatim and n-gram presence in a large web corpus (FineWeb) used as a proxy for web circulation, prefix-completion probes, and the Choice Confusion test. Choice Confusion replaces an item's distractors with correct answers taken from other items; clean models gain from this change, while contaminated models lose. We include Pythia, whose training corpus is public, so that exposure can be checked against actual training text rather than a proxy. We then compare the dialect gap on high-exposure items with the gap on low-exposure items. If $\tau < 1$, the gap should be concentrated in high-exposure items. Part one tells us how large this concentration should be, so part two tests whether the controlled estimate holds in the wild.

The third part generates contamination-free items, which estimates $\Delta_{\text{true}}$ directly. We extend DyVal, which builds fresh arithmetic, logic, and reachability problems from randomly generated graphs, so that each sampled problem is rendered in both SAE and AAE. Two rendering routes are used: the rule pipeline, and a smaller set of renderings written by native speakers, in the style of ReDial's human-annotated pairs. Because the problems did not exist before generation, $G = 0$ for every item, and any remaining gap is the true gap. This part also addresses the gap noted in the file that dynamic evaluation has so far assumed Standard English only.

As a secondary output, we characterize whether Choice Confusion and perplexity probes behave the same way on dialect text. A detector that misfires on AAE simply because the grammar is unfamiliar would contaminate the audit itself, so we calibrate the detectors on the controlled models from part one, where the true contamination status of every item is known.

Baselines

The naive reference point is the raw dialect gap as currently reported. This is the gap obtained by running SAE and AAE versions of a static benchmark and subtracting, with no contamination adjustment. For GPT-4 on MMLU, the Disparities paper reports a 10.7% accuracy drop from SAE to AAE, rising to 18.2% in social sciences and humanities. ReDial reports an average relative drop above 10% and up to 30% on planning. We reproduce the raw gap on the open models under identical prompts and decoding, so that every later adjustment is measured against our own number and not against a different setup. The proposal improves on this baseline if the decomposition shows that $\Delta_{\text{obs}}$ and $\Delta_{\text{true}}$ differ by more than seed-to-seed and item-resampling variance. Showing that they do not differ, with tight intervals, also counts as a result: it would validate the existing literature.

The strongest existing method for separating memorization from reasoning is Yao et al.'s Choice Confusion test combined with their controlled cross-lingual contamination. On MMLU they report a gain of +26.25 points for clean LLaMA3 and a drop of roughly 17 points for contaminated LLaMA3. We run it with their protocol, on the same models and items, and with the same fine-tuning budget as our contamination arms. It is the strongest competitor because it already detects transformed contamination that surface checks miss. Its limitation for this question is that it labels whether an item is contaminated. It does not measure how much of a fairness gap that contamination accounts for, and it has never been validated on dialect text. We beat it if, on the controlled models, our item-level exposure estimates predict the per-item dialect gap better than Choice Confusion labels alone, and if we identify cases where Choice Confusion misreports AAE items.

The strongest existing mitigation is the "Expert Teacher" prompt framing from the Disparities paper, which reduced GPT-4's MMLU gap from 10.7% to 4.8%. We include it under the same prompt budget to ask whether prompting closes the true gap or only the contamination-driven part. Prompting cannot restore memorization on AAE strings, so if the gap that remains after prompting matches $\Delta_{\text{true}}$ from DyVal, prompting is fixing comprehension, not recall.

Two ablations isolate the key ingredient. The first evaluates rule-rendered and human-rendered DyVal items separately, so that artifacts of the rewrite rules are not counted as dialect effects. The second keeps the contamination injection but presents contaminated items with SAE syntax and lightly altered wording. This tells us whether $\tau < 1$ is caused specifically by dialect morphosyntax or would arise from any rewording.

Why not just…?

Why not just translate DyVal into AAE and report the gap? This is the most serious alternative, and it forms part three of the program. On its own, it answers whether a dialect gap exists without contamination. It does not say how much of the already-published gaps were artifacts, and it cannot distinguish a small $\Delta_{\text{true}}$ caused by dialect from one caused by DyVal's formal, template-like style. Without the controlled $\tau$ estimate, a reader cannot transfer the DyVal result to MMLU or ReDial. If DyVal's AAE gap matches the published static gaps, that alternative would have sufficed, and we would report this plainly.

Why not just stratify items by n-gram overlap with web text? The cross-lingual results show that surface detectors miss transformed contamination entirely: MMLU rose by 5 to 10% with no n-gram or perplexity signal. An n-gram split would therefore misclassify many memorized items as clean and bias the estimate of $\tau$ upward. We still run the n-gram split, as one exposure signal in part two, and part one shows how much it under-detects.

Why not just compare items released before and after the model's training cutoff? Chronological splits work for Codeforces, but MMLU, MathQA, and the dialect benchmarks have no post-cutoff counterparts that match in difficulty. Newer items also differ systematically from older ones, which introduces a separate confound. Generated items provide a clean split without this drift.

Why not just standardize the dialect input before answering? ReDial shows that standardization recovers accuracy at the cost of extra tokens and latency, a "dialect tax." The Disparities paper shows that prompting narrows the gap without closing it. These are mitigations; they do not tell us what the gap measures. Our design treats them as conditions to be explained, not as competing answers.

Proposed Experiments and Decisive Checks

The decisive measurement is $\tau$ from the controlled arm, reported with bootstrap intervals over items and fine-tuning seeds. The first check is whether $\tau$ is substantially below the cross-lingual transfer level Yao et al. observed. If dialect rewrites block memorization that survives full translation, the asymmetric-contamination account is supported. The second check is whether, on off-the-shelf models, the dialect gap is concentrated in high-exposure items to the degree that the controlled $\tau$ predicts. The third check is whether $\Delta_{\text{true}}$ on dialect-rendered DyVal, using human renderings, falls below the static MMLU and ReDial gaps by roughly $(1-\tau)G_{\text{SAE}}$. If all three agree, we will have a corrected estimate of how much LLMs genuinely struggle with AAE reasoning, and a reusable correction for future dialect audits. If $\tau \approx 1$ and the gap on DyVal matches the static gap, the published dialect penalties stand as real reasoning deficits, now with contamination ruled out. That outcome is the result the fairness literature has been assuming without evidence.