The prevailing assumption behind the scaling hypothesis is that increasing a model’s size, dataset volume, and computational budget will naturally smooth out its edge cases. Under this view, biases and performance disparities are simply symptoms of underfitting—bugs that will vanish once a model possesses enough parameters to capture the full complexity of human language.
However, empirical evidence across dialects, socio-economic contexts, and languages reveals a different reality. Scaling up large language, vision, and speech models does not inherently resolve performance gaps between dominant and marginalized groups. In fact, it frequently maintains or even widens them.
The Persistence of Dialectal Brittleness
When language models encounter non-standard dialects, such as African-American Vernacular English (AAVE) or South Asian English (SAsE), their accuracy drops sharply compared to their performance on Standard American English (SAE). Crucially, this gap does not close as models grow.
Early investigations by Sophie Groenwold et al. in Investigating African-American Vernacular English in Transformer-Based Text Generation demonstrated clear dialectal performance gaps in early transformer architectures. As models scaled up to frontier-class sizes, the underlying issue remained unresolved. Fangru Lin et al. evaluated state-of-the-art models in Assessing Dialect Fairness and Robustness of Large Language Models in Reasoning Tasks and discovered a profound "dialect brittleness." Even top-tier models experience significant performance drops—often exceeding 10%—on identical reasoning tasks when the prompt shifts from Standardized English to AAVE.
In a parallel single-author study, Fangru Lin confirmed in One Language, Many Gaps: Evaluating Dialect Fairness and Robustness of Large Language Models in Reasoning Tasks that this robustness gap remains relatively constant across model sizes. This persistence indicates that the problem stems from deep-seated data distribution imbalances rather than a lack of raw parameter capacity. The real-world consequence of this invariance is highlighted by Faye Holt et al. in Perceptions of Language Technology Failures from South Asian English Speakers, showing that the performance gap for regional syntactic structures remains consistent across model scales, forcing speakers to actively modify their natural speech to be understood by modern language technologies.
The Scaling-Fairness Paradox
Why doesn't capacity resolve these disparities? The answer lies in how optimization works at scale. As a model expands, it becomes highly efficient at minimizing loss over the dominant statistical patterns in its training data. Instead of learning the nuanced, structural rules of underrepresented dialects, larger models use their increased capacity to overfit to the majority distribution.
This dynamic can cause performance disparities to expand rather than contract. Jamell Dacon et al. demonstrated in Evaluating and Mitigating Inherent Linguistic Bias of African American English through Inference that larger architectures can exhibit a dialect performance gap that is several times wider than that of smaller models.
Average Error Rate
│
│ ▲ (Majority Dialect drops rapidly)
│ │
│ │ ▼ (Minority Dialect drops slowly or plateaus)
│ └────────────────────────►
│ Relative Gap Widens Logarithmically
└──────────────────────────────────► Model Scale (Parameters)
This phenomenon is documented clearly in speech recognition. In Group Fairness in Multilingual Speech Recognition Models, the authors found that as multilingual speech models scale up and improve in average accuracy, the performance disparity between demographic groups grows logarithmically. While the absolute error rate for the best-served population drops rapidly, the relative gap between the best-served and worst-served populations widens, leaving marginalized subgroups behind.
Cross-Modal and Cross-Lingual Invariance
This scale-invariant bias is a structural property of how foundation models represent data across modalities and languages:
- Vision-Language Models: In Bridging the Digital Divide: Performance Variation across Socio-Economic Factors in Vision-Language Models, Joan Nwatu et al. showed that increasing the scale of visual encoders does not resolve performance disparities across socio-economic levels. The relative accuracy gap between recognizing everyday objects from high-income households versus low-income households remains constant across model sizes.
- Multilingual Connectivities: In The Linguistic Connectivities Within Large Language Models, Dan Wang et al. mapped the internal architectures of LLMs and found that while scaling increases the general level of interaction between languages, low-resource or isolated languages remain structurally marginalized regardless of parameter count.
- Cultural Commonsense: Similarly, Lajanugen Logeswaran et al. showed in Understanding the Capabilities and Limitations of Large Language Models for Cultural Commonsense that while scaling improves absolute accuracy on cultural tasks, it fails to reduce the relative performance gap between dominant Western cultures and underrepresented ones.
The root cause across all these domains is that scaling merely allows models to represent skewed training data distributions with higher precision. As Chantal Shaib et al. demonstrated in Learning the Wrong Lessons: Syntactic-Domain Spurious Correlations in Language Models, the tendency of models to rely on syntactic shortcuts and spurious correlations persists from small parameter sizes all the way up to frontier-class models.
While some work, such as Magnus Sahlgren et al. in ELOQUENT 2024 - Robustness Task, indicates that larger models exhibit better robustness against minor stylistic or vocabulary shifts, this localized stability does not automatically translate into demographic equity or structural fairness.
Algorithmic Paths Forward
Because brute-force scaling fails to deliver linguistic equity, addressing performance disparities requires targeted algorithmic and inference-time interventions rather than larger compute budgets.
Modular Adaptation
Instead of forcing a single set of parameters to represent all dialects, researchers have designed modular architectures. Yanchen Liu et al. developed this approach in DADA: Dialect Adaptation via Dynamic Aggregation of Linguistic Rules by training specialized "feature adapters" that capture specific dialectal variations. These adapters are blended dynamically at test time, allowing a base model to achieve linguistic fairness without requiring massive parameter scaling.
Inference-Time Calibration
Many performance gaps in few-shot settings are driven by manageable inference-time biases rather than fundamental capacity limits. In Calibrate Before Use: Improving Few-shot Performance of Language Models, Shi Feng et al. introduced Contextual Calibration. By estimating a model's bias using content-free inputs (like "N/A") and adjusting the output probabilities accordingly, the authors enabled smaller, calibrated models to match or exceed the performance of uncalibrated models many times their size.
Post-Hoc Optimization
To address class-level performance imbalances that persist across scales, researchers also use post-hoc optimization. Ruixi Lin et al. introduced a method in COBias and Debias: Minimizing Language Model Pairwise Accuracy Bias via Nonlinear Integer Programming that reweights output probabilities using discrete optimization. This approach slashes performance disparities while simultaneously boosting overall accuracy.
Together, these developments show that linguistic equity is not an emergent property of scale. Overcoming systemic bias requires changing how models learn and infer from data, rather than simply expanding the size of the network.
Key Foundational Work
Investigating African-American Vernacular English in Transformer-Based Text Generation (2020) — contributed 1 concept elements to this direction.
Sociolectal Analysis of Pretrained Language Models (2021) — contributed 1 concept elements to this direction.
Calibrate Before Use: Improving Few-shot Performance of Language Models (2021) — contributed 1 concept elements to this direction.