Caught in the Web of Words: Do LLMs Fall for Spin in Medical Literature?

Hye Sun Yun, Karen Y. C. Zhang, Ramez Kouzy, Iain James Marshall, Junyi Jessy Li, Byron C. Wallace

Abstract

As Large Language Models (LLMs) become central to medical research synthesis and clinical decision support, their ability to remain objective in the face of biased reporting is critical. This paper investigates the susceptibility of 22 LLMs to "spin"—linguistic phrasing that overstates the benefits of a treatment despite non-significant primary results. Our findings reveal a "Detection-Interpretation Paradox": while advanced models can identify spin with high accuracy, they remain significantly more susceptible to its influence than human experts when interpreting clinical data. We evaluate these models across detection, interpretation, and propagation tasks, and demonstrate that specific prompting strategies can mitigate, but not entirely eliminate, this bias.

Introduction

In the realm of medical literature, the integrity of an abstract is paramount. Clinicians often rely on these summaries to guide practice, yet abstracts are frequently subject to "spin." Spin refers to reporting strategies—intentional or otherwise—that highlight secondary benefits or "trends toward significance" when the primary endpoints of a study fail to reach statistical significance.

With the advent of LLMs as tools for "trawling" vast quantities of medical data, there is a growing risk that these models might not only fail to recognize spin but may also amplify it when generating summaries for patients or clinicians. This study seeks to determine if LLMs can see through the "web of words" or if they are inherently more gullible than seasoned medical professionals.

Methodology

The researchers conducted a comprehensive evaluation of 22 LLMs, including proprietary models (e.g., GPT-4o, Claude 3.5 Sonnet) and specialized biomedical models (e.g., Med42, BioMedGPT). The evaluation was structured around three primary tasks:

  1. Spin Detection: A binary classification task where models identified whether a given abstract contained spin.
  2. Result Interpretation: Models were asked to rate the benefit of a treatment on a 0–10 scale. This was tested using a controlled A/B setup: models read either a "spun" version of an abstract or a "neutralized" version of the same study where the data remained identical but the biased phrasing was removed.
  3. Spin Propagation: Models generated Plain Language Summaries (PLS) intended for a 5th-grade reading level to determine if the bias from the technical abstract was carried over into simplified patient-facing content.

Dataset

The study utilized a high-quality, manually curated dataset from Boutron et al. (2014), consisting of 60 abstracts from cancer-related Randomized Controlled Trials (RCTs). This dataset provided 30 pairs of original "spun" abstracts and their expert-neutralized counterparts, allowing for a precise measurement of how phrasing alone influences model output.

Key Results and Analysis

The Detection-Interpretation Paradox

The study identified a striking disconnect between a model's ability to recognize bias and its ability to resist it. Claude 3.5 Sonnet, for instance, achieved a near-perfect detection accuracy of 97%. However, even when models correctly identified that an abstract contained spin, their interpretation of the treatment's benefit remained skewed by that same spin. This suggests that the linguistic framing of a study exerts a "priming" effect that persists even through the model's reasoning layers.

AI Gullibility vs. Human Expertise

When comparing LLMs to human experts, the gap in skepticism was profound. On the 10-point interpretation scale, human experts showed a mean difference of only 0.71 between spun and neutral abstracts, demonstrating a professional capacity to focus on the underlying data. In contrast, the LLM group showed a mean difference of 3.38—nearly five times higher than humans. The models were significantly more likely to "believe the hype" presented in the spun text.

Propagation and Safety Concerns

The study found that spin is "contagious." When LLMs simplified technical abstracts into plain language, they implicitly incorporated the original spin into the summaries. This poses a significant safety risk for patient-facing AI applications, as it could lead laypeople to perceive a treatment as highly effective when the clinical evidence is actually inconclusive.

Generalist vs. Specialist Models

Interestingly, specialized biomedical models did not provide a safeguard against spin. General-purpose models like Claude 3.5 and GPT-4o generally outperformed specialized models like BioMedGPT in both detection accuracy and resistance to interpretation bias. This suggests that the reasoning capabilities of larger generalist models are currently more effective at handling complex linguistic nuances than smaller, domain-specific models.

Mitigation and Future Work

The researchers tested several mitigation strategies, finding that "joint prompting" was the most effective. By forcing the model to first perform a spin detection task before interpreting the results, the interpretation bias was reduced from a baseline of 3.38 to 1.14. While this brings AI closer to human-level skepticism, a gap still remains.

Future research is required to expand the dataset beyond oncology and to investigate whether fine-tuning models specifically on "de-spinning" tasks can provide a more permanent solution. Additionally, the impact of AI-propagated spin on human decision-making—where a human user reads an AI-generated summary—remains a critical area for investigation.

Conclusion

LLMs are powerful tools for medical information retrieval, but they are not yet objective arbiters of truth. Their susceptibility to linguistic spin exceeds that of human experts, and their tendency to propagate this bias into simplified summaries necessitates caution. For now, the "web of words" remains a potent trap for even the most advanced artificial intelligence.

Summary updated

BibTeX
@inproceedings{yun2025caught,
  author = {Hye Sun Yun and Karen Y. C. Zhang and Ramez Kouzy and Iain James Marshall and Junyi Jessy Li and Byron C. Wallace},
  title = {{Caught in the Web of Words: Do LLMs Fall for Spin in Medical Literature?}},
  booktitle = {CHIL},
  year = {2025},
  url = {https://openreview.net/forum?id=nz3OfZ1Gmh},
}
Figure