The Correctness Bias in Large Language Models

16 concepts8 papers1 proposal

We spend immense computational resources training Large Language Models (LLMs) to be accurate, helpful, and factually correct. Through massive pre-training on human knowledge and subsequent alignment via reinforcement learning, these models learn to treat "truth" as a steep gravity well.

However, this relentless optimization introduces a systemic blind spot: the correctness bias. When an application requires a model to intentionally generate or evaluate plausible incorrectness—such as creating multiple-choice distractors, synthesizing realistic fake news, or diagnosing student errors—the model's internal drive toward factual accuracy actively works against it. Understanding why this bias occurs and how it disrupts downstream tasks is essential for building robust evaluation and training pipelines.

The Generation Bottleneck: Failing to Lie Plausibly

To generate a high-quality "negative example" (like a wrong answer on a test or a piece of propaganda), a model must do something surprisingly difficult: it must deviate from the truth without collapsing into obvious nonsense. It needs to navigate a narrow corridor of plausible falsehood. Because standard language models are optimized to predict the most historically and logically consistent tokens, they struggle to break away from factual trajectories.

When prompted to generate deceptive or hallucinated text, LLMs typically fail in one of two ways:

  1. They default back to the truth: As observed by Shouqiang Liu et al. in DGRC: An Effective Fine-tuning Framework for Distractor Generation in Chinese Multi-choice Reading Comprehension, pre-trained models frequently default to accuracy or accidentally output the correct answer even when explicitly instructed to generate a wrong alternative.
  2. They resort to lazy paraphrasing: Instead of inventing novel, structurally sound errors, models often produce text that remains too semantically or logically tied to the source material. Prasetya Ajie Utama et al. demonstrated this limitation in Falsesum: Generating Document-level NLI Examples for Recognizing Factual Inconsistency in Summarization, noting that generative models exhibit a strong bias toward copying source content even when tasked with creating extrinsic hallucinations. Similarly, Luise Dürlich et al. found in Overview of the CLEF-2024 Eloquent Lab: Task 2 on HalluciGen that models struggle to intentionally generate plausible hallucinations for adversarial testing, producing outputs that are far too factually grounded to serve as effective test cases.

This inability to break free from the truth hampers our ability to generate high-quality training data. For instance, creating robust disinformation detectors requires training data that mirrors the sophisticated, truth-twisting strategies of human bad actors. As argued by Heng Ji et al. in Faking Fake News for Real Fake News Detection: Propaganda-Loaded Training Data Generation, overcoming the model's inherent bias toward factual accuracy or simple paraphrasing is the primary hurdle to generating realistic, propaganda-loaded training sets. A similar bottleneck occurs in multimodal domains, where the authors of Mitigating Object and Action Hallucinations in Multimodal LLMs via Self-Augmented Contrastive Alignment point out that relying on standard model generations fails to capture the nuanced negative examples required to train models against visual hallucinations.

[Standard Optimization] ---> Steep path to Truth/Correctness (High Probability)
                                 |
                                 v 
[Distractor Generation] ---> Model struggles to step away from the truth;
                             either falls back to the correct answer or 
                             creates an obvious, unplausible error.

The Evaluation Paradox: Blind Judges of Error

The correctness bias does not just impair generation; it also blinds LLMs when they act as evaluators or judges. When an LLM is tasked with grading a multiple-choice question's distractors, it encounters an incorrectness paradox: the model's internal evaluation metrics are tuned to penalize falsehood, making it fundamentally unsuited to recognize the utility of a good false statement.

In D-GEN: Automatic Distractor Generation and Evaluation for Reliable Assessment of Generative Models, Grace Byun et al. show that general-purpose LLM judges routinely fail to assess distractor quality because they penalize options simply for being factually incorrect. They overlook the core requirement of a distractor: it must be false, but it must remain contextually plausible enough to challenge a human or a model's reasoning.

Furthermore, when models encounter incorrect information in context, their correctness bias can morph into a conformity bias. In Scoring Verifiers: Evaluating Synthetic Verification for Code and Reasoning, Aleksander Ficek et al. discovered that when LLMs are provided with potentially incorrect code solutions during test-case generation, they exhibit a strong bias toward replicating and conforming to those very errors, rather than critically evaluating and exposing them.

This creates a significant barrier for AI in education. As highlighted by Shashank Sonkar et al. in MalAlgoQA: Pedagogical Evaluation of Counterfactual Reasoning in Large Language Models and Implications for AI in Education, because evaluation frameworks focus almost exclusively on generating correct answers, LLMs struggle to diagnose why a student might be wrong. They lack the counterfactual reasoning capacity to navigate "malgorithms"—logically coherent but fundamentally flawed reasoning paths.

Decoupling Truth from Structure

To overcome the correctness bias, researchers must find ways to decouple a model's internal knowledge of the truth from its text-generation mechanics. If a model tries to generate a wrong answer and a plausible narrative simultaneously, the gravity of the correct answer pulls the token probabilities away from the desired distractor.

A powerful intuition for solving this is to force the model to anchor its understanding of the truth before attempting to deviate from it. For example, the framework proposed by Shouqiang Liu et al. utilizes a "Hard Chain-of-Thought" mechanism. Instead of asking the model to generate a distractor directly from a prompt, the model is forced to solve the reading comprehension question correctly first. By explicitly mapping out the correct reasoning path in its context window, the model can then deliberately and systematically alter specific logical steps to construct a false alternative that remains highly relevant to the text.

By treating the generation of falsehood not as a lack of knowledge, but as a controlled, counterfactual deviation from a known truth, recent frameworks are beginning to bypass the correctness bias—allowing LLMs to effectively model the full spectrum of human error and deception.

Key Foundational Work

Faking Fake News for Real Fake News Detection: Propaganda-Loaded Training Data Generation (2023) — contributed 1 concept elements to this direction.
Falsesum: Generating Document-level NLI Examples for Recognizing Factual Inconsistency in Summarization (2022) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_correctness_bias_6a6151fb,
  title = {correctness bias},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/correctness-bias/rdr_6a6151fbaad8465282750df49cd6dddc},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_correctness_bias_6a6151fb_v1,
  title = {correctness bias},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/correctness-bias/rdr_6a6151fbaad8465282750df49cd6dddc/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake