Large language models answer the same question worse when it is asked in a non-standard dialect. On parallel reasoning queries written by native speakers of African American Vernacular English (AAVE), even GPT-4o loses 11.6% accuracy relative to the Standardized English version, and scale does not close the gap. Existing fixes either change the prompt or translate the query first, and both treat dialects as a handful of fixed bins that are optimized on average. We propose a different kind of fix. We compute one linear edit of the model's hidden states that shrinks the directions along which dialect rewrites of the same question drift away from the standard version. The edit is chosen to help the worst-served dialect, not the average one, and it is constrained to leave task content intact.
Choosing that edit is naturally a semidefinite program, meaning an optimization over a symmetric matrix that is required to have no negative eigenvalues. Here the matrix lives in the model's 4096-dimensional hidden space, which is where the scalability research becomes the enabling ingredient rather than decoration. The program has only as many group constraints as there are dialects or dialect sites (9 to about 220), while the matrix is huge. This is exactly the regime where the optimum is provably low-rank and where low-rank first-order solvers can replace interior-point methods that would stall at cubic cost per step. The deliverable is an edit with a certificate: a solver-reported bound on how far the worst dialect's residual drift is from the best achievable under the content budget.
Core Claim
Our claim is that a max-min (worst-group) linear edit of hidden states, solved as a semidefinite relaxation with low-rank methods, reduces the worst-dialect accuracy gap more than two alternatives: the pooled, average-case version of the same edit, and prompt-level robust selection. It should also do this at a runtime that is feasible for full-width models. The comparison with the pooled edit is the heart of the proposal. If worst-group optimization does not beat pooled optimization on the worst dialect, the semidefinite machinery has not earned its place.
Concretely, take a layer $\ell$ of a frozen model and a set of dialect groups $g = 1,\dots,G$. From parallel data we form a drift covariance $D_g = \mathbb{E}[(h_g - h_{\mathrm{std}})(h_g - h_{\mathrm{std}})^\top]$, where $h_g$ and $h_{\mathrm{std}}$ are the mean-pooled prompt hidden states of the dialect and standard versions of the same item. We also form a content covariance $C$ of standard-version hidden states around their mean. We seek a symmetric matrix $P$ and use it as the soft edit $h \mapsto h - P(h - \mu)$ on prompt tokens, where $\mu$ is the mean standard-version hidden state. The program is
$$\max_{P,\,t}\; t \quad \text{s.t.}\quad \langle D_g, P\rangle \ge t \;\; (g=1,\dots,G),\quad \langle C, P\rangle \le \varepsilon\,\mathrm{tr}(C),\quad \mathrm{tr}(P)=k,\quad 0 \preceq P \preceq I .$$
In words, $P$ picks out roughly $k$ directions. It captures as much drift as possible for the dialect whose drift is least captured, while removing at most a fraction $\varepsilon$ of content variance. The constraint set $0 \preceq P \preceq I$ with fixed trace is the convex hull of rank-$k$ projections. Every feasible $P$ is therefore directly usable as a shrinkage, and no rounding step is needed.
The program has $m = G + 2$ scalar constraints beyond the eigenvalue bounds. The Pataki–Barvinok counting argument, the same one behind the rank threshold $p(p+1)/2 > m$ in The non-convex Burer-Monteiro approach works on smooth semidefinite programs, bounds how many eigenvalues of an extreme optimal $P$ can lie strictly between 0 and 1, and that bound grows only like $\sqrt{2m}$. With 9 dialects this is about 5 extra directions. With 220 dialect sites it is about 21, still tiny next to a hidden width of 1280 or 4096. The optimum is therefore close to a low-rank projection, and a solver can keep compact iterates.
Relation to Prior Work
The motivating evidence comes from three evaluation papers. One Language, Many Gaps builds ReDial, 1,216 parallel AAVE/Standardized English reasoning queries, and shows drops above 10% that reach about 30% on multi-step planning. It also shows that translating to standard English never recovers the standard baseline. MDial provides parallel dialogues across 9 English dialects. It shows that models misidentify dialects (under 70% accuracy), and that most grammatical features should be understood but not reproduced. That asymmetry is why our edit acts only on prompt tokens. Dialetto, ma Quanto Dialetto? shows that speech-recognition error rises smoothly with phonetic distance from Tuscan across 223 Italian sites. This motivates treating many sites as groups instead of a few coarse bins.
The closest method is Equitable System-Prompt Selection via Constrained Mixed-Strategy GroupDRO. It mixes over a pre-scored pool of system prompts to minimize worst-group loss under a mean-loss budget, and it solves this as a linear program. We borrow its logic of optimizing the worst group under an average-performance budget. We differ in what is optimized: a continuous edit of internal representations instead of weights over a fixed prompt pool. That turns the linear program into a semidefinite one and removes the need to score every prompt against every group.
The moment-based robust optimization papers show that second-moment uncertainty leads to semidefinite duals, for example Distributionally Robust Risk Map and Max-Min Bilinear Completely Positive Programs. They supply the modeling pattern but have never been used on language representations. We deliberately use a simpler, finite-group max-min program rather than a full covariance ambiguity set. This keeps the constraint count small, which is the property that makes low rank possible.
The solver comes from the scalability line, and one prior result constrains it. The Burer–Monteiro landscape guarantees require smooth equality-constrained sets, and our upper bound $P \preceq I$ violates that assumption. We therefore do not claim those guarantees. Our primary solver is the approach of A conditional gradient homotopy method with applications to semidefinite programming: each step over the fixed-trace set with eigenvalues in $[0,1]$ needs only a top-$k$ eigenvector computation, and group constraints are handled by a homotopy penalty. As a heuristic comparison we run a Burer–Monteiro-style factorization $P = YY^\top$ that projects onto the eigenvalue box. The interior-point analyses in A Faster Interior Point Method for SDP and Solving SDP Faster establish the $O(mn^3)$-type per-step cost that we expect to be infeasible at $n = 4096$ across many layers, and we test this directly. We do not use the GNN surrogates of On the Expressive Power of GNNs to Solve Linear SDPs. Their $O(n^3)$ per-iteration cost gives no advantage in our regime.
Research Program
The first component is the estimator. For open-weight models (Llama-3-8B and 70B, both tested in ReDial), we extract prompt hidden states at every fourth layer for all parallel items. We then estimate $D_g$ and $C$ with shrinkage, because some groups have only a few hundred items. The second component is the solver study. We run the conditional gradient homotopy method and the factorized heuristic, and compare them against SCS and the sparse homogeneous-embedding ADMM of Fast ADMM for homogeneous self-dual embedding of sparse SDPs. We report wall-clock time, memory, duality gap, and the rank of the returned $P$ against the $\sqrt{2m}$ prediction.
The third component is intervention. We apply $h \mapsto h - \alpha P(h-\mu)$ on prompt tokens at a chosen layer, with layer and $\alpha$ selected on a validation split, and leave generation untouched. The fourth component is generalization across a continuum. On the Vivaldi corpus we apply the same edit to Whisper-large-v3 encoder states, where the 15 sentence recordings per site are parallel across sites. We fit on a subset of sites and evaluate on held-out sites. This tests whether a worst-group edit trained on sampled sites protects unsampled peripheral ones.
Baselines
The naive reference point is the unedited frozen model under zero-shot prompting. On ReDial, One Language, Many Gaps reports an 11.6% zero-shot drop from standard English to AAVE for GPT-4o. For the open models we will measure our own unedited gap on the same splits, because the file does not give per-model numbers for them. On Vivaldi, the reported Whisper-large-v3 median chrF2 is 62.32 for Tuscan versus 23.69 for Sardinian and 26.40 for Piedmontese, which serves as the unedited reference for worst-site performance.
The key ablation, and in our view the most dangerous competitor, is the pooled edit. It solves the same program with the per-group constraints replaced by a single constraint on the average drift $\bar D = \frac{1}{G}\sum_g D_g$. This reduces to a generalized eigenproblem with a closed-form solution. It uses the same data, layer search, $k$, $\varepsilon$, and $\alpha$ grid. No reported result for this was found on Lacuna. The proposal beats it only if worst-group accuracy (or worst-site chrF2) improves by a margin that exceeds the variation across seeds and bootstrap resamples, while standard-English accuracy stays within the same budget.
The strongest existing method is constrained mixed-strategy GroupDRO over prompt pools from Equitable System-Prompt Selection. It reports average reductions of 13.1% in overall mean loss, 13.2% in worst-25% mean loss, and 13.7% in absolute worst-case loss relative to no mitigation on its consumer finance benchmark. We run it on ReDial and MDial with a 30-prompt pool, inside the 25–35 range where that paper reports most of the benefit, scored on the same training split. We also run our edit on the consumer finance benchmark, whose variants cross language and literacy signals, so that the comparison happens on that method's home ground. Beating it means a larger relative reduction in worst-group loss at an equal or smaller increase in mean loss.
We also compare against the translate-then-answer pipeline, with its token and latency cost reported, and against five-shot in-dialect prompting, which AL-QASIDA found helps substantially. Finally, we compare against LoRA fine-tuning on the same parallel training pairs with a matched number of trainable parameters where possible.
Why not just…?
Why not just take the average-drift directions and project them out? This is the obvious closed-form alternative, and it may be enough. If one dialect such as Nigerian English dominates the pooled drift, the pooled edit will spend its $k$ directions there and neglect dialects whose drift is smaller but harmful, like Canadian English, which MDial shows is hardest to identify. Whether this happens is an empirical question, and the pooled-edit ablation settles it. If the worst-group program's advantage is within noise, we will report that the semidefinite formulation is unnecessary for this problem.
Why not just use the prompt-mixture linear program, which already works? It optimizes over a fixed pool, so it cannot help any dialect that no prompt in the pool happens to serve. It also cannot scale to hundreds of site-level groups without scoring every prompt against every site. Its 13.7% worst-case gain is the bar we must clear on its own benchmark. If it matches us there and on ReDial, the representation edit adds nothing beyond cost.
Why not just translate to the standard dialect first? ReDial reports that translation never reaches the standard baseline and adds a token and latency tax. We will measure both on the same items, so the reader can see accuracy against cost for the two approaches together.
Why not just fine-tune on parallel dialect data? This is a real threat, and it may well beat a single linear edit on raw accuracy. The edit's claimed advantages are that it touches no weights, costs only a matrix multiply at inference, and carries a certificate on the worst group. We will report accuracy for both, and also the standard-English regression and generation-mirroring rate on MDial, where fine-tuning on dialect text may push outputs toward reproducing user morphosyntax.
Why not just run gradient descent on the max-min objective over orthonormal bases? It could find similar solutions. It gives no global certificate, however, and nonsmooth max-min problems are prone to poor local solutions. We run it with the same budget and report both its worst-group value and the convex bound. If the two match, this simpler solver is what a practitioner should use.
Proposed Experiments
The first decisive check is on ReDial with Llama-3-8B and 70B, using 5-fold splits over items. The primary metric is the worst-dialect accuracy gap. Because ReDial has a single dialect pair, it is really the AAVE gap here, so the max-min and pooled versions coincide. That makes ReDial a check that the edit helps at all, not a test of worst-group optimization. The worst-group test is on MDial with 9 dialects, where we report worst-dialect QA accuracy, standard-English accuracy, and the rate at which responses reproduce dialect grammatical features that should not be mirrored.
The continuum test is on Vivaldi with held-out sites. We report worst-decile chrF2 and the correlation between error and phonetic distance, which is -0.5 to -0.58 unedited. We will count the edit as flattening the continuum if that correlation weakens on held-out sites.
The scalability test sweeps hidden width (1280, 4096, 8192) and group count (9 to 220). It reports solver time, memory, duality gap, and returned rank, and checks whether rank stays near $k + \sqrt{2m}$ while SCS and ADMM time grow much faster.
The proposal fails cleanly if either of two things happens. Worst-group editing may not beat the pooled edit on MDial or Vivaldi held-out sites. Or the content budget that preserves standard-English accuracy may leave too little drift removed to move the AAVE gap on ReDial.
References
- The non-convex Burer-Monteiro approach works on smooth semidefinite programs
- One Language, Many Gaps: Evaluating Dialect Fairness and Robustness of Large Language Models in Reasoning Tasks
- Should LLMs, like, Generate How Users Talk? Building Dialect-Accurate Dialog[ue]s Beyond the American Default with MDial
- Dialetto, ma Quanto Dialetto? Transcribing and Evaluating Dialects on a Continuum
- Equitable System-Prompt Selection via Constrained Mixed-Strategy GroupDRO
- Distributionally Robust Risk Map for Learning-Based Motion Planning and Control: A Semidefinite Programming Approach
- Max-Min Bilinear Completely Positive Programs: A Semidefinite Relaxation with Tightness Guarantees
- A conditional gradient homotopy method with applications to Semidefinite Programming
- A Faster Interior Point Method for Semidefinite Programming
- Solving SDP Faster: A Robust IPM Framework and Efficient Implementation
- On the Expressive Power of GNNs to Solve Linear SDPs
- Fast ADMM for homogeneous self-dual embedding of sparse SDPs
- AL-QASIDA: Analyzing LLM Quality and Accuracy Systematically in Dialectal Arabic