When evaluating large language models (LLMs) on safety and fairness, standard benchmarks often present clean, straightforward prompts. Under these ideal conditions, modern models appear remarkably unbiased. However, this neutrality is frequently an artifact of low task complexity. As tasks become more structurally, logically, or contextually demanding, a model's capacity to maintain objective reasoning degrades, causing it to fall back on statistical shortcuts, demographic stereotypes, and superficial heuristics.
Understanding why complexity induces bias requires looking at how language models process information when their reasoning capabilities are pushed to their limits.
The Fallback to Heuristics under Cognitive Load
In human psychology, when individuals face high cognitive load or ambiguous situations, they often rely on mental shortcuts or implicit biases. A similar failure mode occurs in deep learning architectures. When the logical or syntactic complexity of a prompt increases, the model's structured reasoning path breaks down, and it defaults to the strongest statistical associations present in its training data.
This phenomenon is clearly visible across different dimensions of complexity:
- Syntactic Complexity: In coreference resolution and machine translation, as the syntactic distance between entities grows or the density of pronouns increases, models struggle to maintain structural tracking. Shahar Levy et al. demonstrated this behavior using the BUG dataset in Collecting a Large-Scale Gender Bias Dataset for Coreference Resolution and Machine Translation, showing that models consistently fall back on gender stereotypes when linguistic structures become convoluted.
- Prompt and Relationship Complexity: Biases can remain hidden in simple settings but surface when tasks scale. By systematically varying prompt difficulty across dimensions like size, variety, and relationship complexity, the LLM Stereotype Index (LSI) introduced by Hari Shrawgi et al. in Uncovering Stereotypes in Large Language Models: A Task Complexity-based Approach revealed that advanced models like GPT-4 exhibit significantly more pronounced demographic biases as task demands scale up.
- Logical Complexity: When the underlying reasoning graph of a problem is manipulated to be more intricate, models fail to maintain objective logic. In DARG: Dynamic Evaluation of Large Language Models via Adaptive Reasoning Graph, Zhehao Zhang et al. showed that increasing the logical complexity of social reasoning tasks causes models to abandon objective reasoning and revert to learned social stereotypes.
The Expansion of Linguistic Surface Area
Complexity does not only manifest as abstract logical difficulty; it also manifests physically as text length and structural variety, often referred to as the "linguistic surface area."
When a model evaluates or generates longer, more complex responses, the increased surface area provides substantially more opportunities for subgroup-specific patterns to trigger errors. In automated grading systems, for instance, longer essays or spoken responses contain a wider variety of syntactic choices and structural patterns.
As explored by Alexander Kwako et al. in Does BERT Exacerbate Gender or L1 Biases in Automated English Speaking Assessment?, this expanded surface area makes it difficult for scoring models to separate true language proficiency from demographic-linked language variations. The model essentially gets lost in the complex linguistic details, allowing demographic-specific patterns to skew its final output.
Contextual Complexity and Advanced Reasoning Models
Counterintuitively, superior reasoning capabilities do not make a model immune to complexity-induced bias. In fact, when models act as judges or evaluators, advanced reasoning capabilities can make them more susceptible to subtle contextual biases.
When evaluating complex tasks, human designers often provide models with auxiliary information, such as grading rubrics or reference answers. However, as the task's core reasoning demands increase, the model becomes highly sensitive to any subtle biases embedded within this guiding context.
In Curse of Knowledge: Your Guidance and Provided Knowledge are biasing LLM Judges in Complex Evaluation, Weiyuan Li et al. introduced the ComplexEval benchmark to highlight this exact vulnerability. The authors found that the severity of biases introduced by auxiliary information scales directly with task complexity. Surprisingly, advanced reasoning models experienced disproportionately larger accuracy drops in these complex, biased evaluation contexts than standard, general-purpose LLMs.
A Boundary Condition: Internal vs. External Bias
While internal biases (like stereotypes) emerge when a model is overwhelmed by complexity, external biases (like direct prompt manipulation) operate under a different dynamic.
In Investigating the Effects of Cognitive Biases in Prompts on Large Language Model Outputs, Yan Sun et al. observed that simpler reasoning tasks are actually more vulnerable to direct linguistic nudges (e.g., adding "I think the answer is B" to a prompt) than complex ones. This suggests an important distinction:
- High-complexity tasks exhaust a model's reasoning capacity, causing it to default to its internal trained stereotypes and heuristics.
- Low-complexity tasks require so little processing depth that shallow, external linguistic cues can easily hijack the model's final answer.
The Human Parallel: Complexity Bias in Evaluation
This relationship between complexity and bias is not unique to silicon. Human evaluation frameworks suffer from a remarkably similar structural flaw.
When human crowdworkers are tasked with evaluating machine translation quality, the length of the sentence acts as a primary driver of cognitive load. In Device-Type Influence in Crowd-based Natural Language Translation Tasks, Michael Barz et al. demonstrated that longer sentences consistently receive lower quality ratings from humans regardless of their actual linguistic accuracy. The sheer cognitive strain of processing longer text segments introduces an evaluative bias, making it difficult to isolate true system errors from the inherent difficulty of the assessment task.
Whether in humans or language models, complexity acts as a solvent for objective reasoning. To build truly robust AI systems, evaluation frameworks must move away from static, simple test cases and explicitly measure how model fairness scales under intense cognitive and structural load.
Key Foundational Work
Device-Type Influence in Crowd-based Natural Language Translation Tasks (short paper) (2018) — contributed 2 concept elements to this direction.
Collecting a Large-Scale Gender Bias Dataset for Coreference Resolution and Machine Translation (2021) — contributed 1 concept elements to this direction.