Framing-Induced Vulnerability in Large Language Models: Evidence from Chinese News Headlines

Mo Zhou, Yiwei ZhangarXiv 2026 · arXiv

TL;DR

Large Language Models (LLMs) are increasingly acting as "evaluative gatekeepers," deciding which news stories are severe or misleading. This paper demonstrates that these models are highly vulnerable to framing effects: simply changing the wording of a headline—without changing the underlying facts—significantly biases an LLM's judgment of an event's severity. While providing the full article body mitigates this bias by 54–80%, a substantial "robustness gap" remains.

What Problem Does This Paper Address?

We often think of LLMs as neutral tools for summarizing or classifying text. However, in modern news pipelines, they function as gatekeepers—ranking, routing, and moderating content before it ever reaches a human reader. If a model’s judgment is swayed by how a fact is presented (framing) rather than the fact itself, it introduces a systematic bias into our information ecosystem.

While framing effects in humans are well-documented (e.g., being more likely to support a policy framed as having a "90% success rate" vs. a "10% failure rate"), it was unclear if LLMs exhibit similar vulnerabilities. Furthermore, previous research often relied on artificial laboratory examples. This paper bridges the gap by auditing real-world news ecology and testing whether providing full article context can "cure" this framing sensitivity.

Methods

The researchers conducted two linked studies:

  1. Ecological Audit: They collected and analyzed over 63,000 Chinese news headlines to understand how framing is actually used by major outlets. They developed a five-level coding scheme ranging from "L1: Objective description" to "L5: Multiple simultaneous manipulations."
  2. Controlled Experiment: They selected 60 events (30 real, 30 synthetic) and wrote five matched headline variants for each, ranging from objective baselines to emotionally intensified or positively reframed versions. They then tested four prominent Chinese LLMs—DeepSeek-v4-flash, MiMo-v2.5-pro, Dots3-note-preview, and LongCat-2.0—asking them to rate the "misleadingness" of the headline and the "severity" of the event.

Finally, they tested a "mitigation" strategy: giving the model the full article body alongside the headline to see if the additional context reduced the framing bias.

Datasets

  • Chinese News Headline Corpus: A newly introduced dataset of 63,926 deduplicated headlines from Xinhua, China News Service, and Jingchu Net (collected between January and September 2026).
  • Framing Annotations: The corpus includes a five-level framing intensity annotation.
  • Experimental Set: 60 "fact cards" (verified against authoritative sources) with five manually written headline variants each, plus 29 naturally occurring objective/manipulative headline pairs.

Key Insights and Ideas

  • Universal Bias Patterns: Surprisingly, all four independently developed model families showed the exact same direction and ordering of shifts in judgment. This suggests that framing vulnerability might be a fundamental property of how current LLMs process language, likely inherited from human-centric training data.
  • "Numerical Shock" is King: The audit revealed that "numerical shock" (using numbers to create an emotional reaction) is the most common manipulative device in Chinese news, present in over 20% of headlines.
Figure 2: Manipulative devices in the headline corpus. Shares are computed over all 63,926 coded headlines; 36.1% carried no identifiable device and the remainder show the most frequent single devices.
  • Severity is Malleable: Framing doesn't just change whether a headline looks like clickbait; it changes the model's perception of the event itself. An emotionally charged headline makes the model believe the underlying incident is more serious than an objective headline describing the same facts.
  • Context is a Shield, Not a Cure: Providing the article body is a "cheap" and effective mitigation, but it doesn't eliminate the bias. The model's judgment is still anchored by the headline.

Key Results

  • Framing Shifts Ratings: Emotionally intensified headlines (H3) raised severity ratings by 0.65 to 0.88 points on a 5-point scale. Conversely, positive reframing (H4) lowered them by 0.34 to 0.52 points.
  • Consistent Misleadingness: The models consistently rated manipulative frames as more misleading, with a stable ordering across all models: Negative Verb ≈ Emotion+Number > Positive > Objective > Passive.
Figure 4: Misleadingness ratings by headline variant. Lines give model means on the five-point scale. The dashed segment gives the human reference contrast between H1 and H2 for the 30 real events (n = 20 raters, 600 ratings).
  • Significant Mitigation: Adding the article body reduced the severity bias by 54% to 80% across the models. As shown below, the "offset" from the objective baseline drops sharply when context is added, though it remains positive.
Figure 5: Severity offset from the objective baseline, with and without the article body. Labels on the right give the bias reduction achieved by adding body text. All four reductions were significant at p < .01.

Limitations and Future Work

  • Language Specificity: The study focused exclusively on Chinese news and Chinese-developed models. Whether these specific framing patterns hold across other languages and cultures remains an open question.
  • Causal Attribution: The headline variants often changed multiple linguistic features at once (e.g., adding an emotional verb and an approximate number). Future work could isolate these variables to see which specific "nudges" are most powerful.
  • Model Evolution: The study used a 2026 model snapshot. As models evolve, their sensitivity to framing may change, necessitating ongoing "robustness checks" as part of standard deployment governance.

Summary updated

Abstract

Large language models increasingly evaluate, route, and moderate news, so a model's judgment of a headline has become a gatekeeping decision. We ask whether that judgment follows the fact being reported or the way the fact is framed. We combine an ecological audit of 63,926 Chinese news headlines with a controlled experiment. The experiment covers 30 real events, 30 synthetic events, and 29 natural headline pairs, tested on four Chinese models. Changing only the headline frame shifted both perceived misleadingness and event severity, and the direction and ordering of the shifts were the same for DeepSeek, MiMo, Dots, and LongCat. Emotionally intensified headlines raised severity ratings by 0.65 to 0.88 points, whereas positively reframed headlines lowered them by 0.34 to 0.52 points. Supplying the article body reduced this severity bias by 54 to 80 percent, and all four reductions were significant at p < .01. A small human reference study showed the same direction of effect. Headline-only LLM evaluation therefore carries a measurable robustness gap, and framing sensitivity belongs among the reliability properties that deployment and governance check.
BibTeX
@misc{zhou2026framing,
  author = {Mo Zhou and Yiwei Zhang},
  title = {{Framing-Induced Vulnerability in Large Language Models: Evidence from Chinese News Headlines}},
  year = {2026},
  eprint = {2611.13531},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.13531},
}
Figure