Same Question, Different Answers: Auditing Persona-Dependent Variation in Six LLMs on Smoking-Cessation Advice

Artin Yousefi, Aizierjiang Aiersilan, Lorien C Abroms, David A BroniatowskiarXiv 2026 · DOI · arXiv

TL;DR

This study audits six major LLMs to reveal that smoking-cessation advice varies significantly based on the user's stated persona, with models frequently withholding medical facts from teens and self-described experts. The researchers demonstrate that a "factual floor" system prompt can successfully narrow these disparities, though the intervention's impact on overall response quality is highly model-dependent.

What Problem Does This Paper Address?

As millions of people turn to AI chatbots for health information, a critical question arises: is the advice consistent, or does the model's "sycophancy"—its tendency to mirror the user's stated identity—lead to biased or incomplete medical information?

The authors tackle this in the context of smoking cessation, a field with clear, codified clinical guidelines (e.g., from the U.S. Preventive Services Task Force). While LLMs are known to adapt their tone to users, this paper investigates whether they also vary the substance of medical facts provided. Specifically, do models withhold evidence-based treatments from certain groups, such as adolescents or those who claim to be medical experts?

Methods

The researchers conducted a multi-model audit across six commercially deployed LLMs from four providers: Anthropic (Claude Haiku/Sonnet), Google (Gemini Flash/Pro), OpenAI (GPT-5.5), and xAI (Grok 4.3).

The experimental design involved:

  1. Persona Construction: 12 synthetic personas spanning four life stages (Teen, College, Mid-career, Retired) and three expertise levels (Novice, Informed, Expert).
  2. Querying: Each persona asked four questions (e.g., "How do I quit smoking?") across two conditions: a Baseline (no instructions) and a System Prompt condition.
  3. The "Factual Floor" Intervention: The authors introduced a short system prompt requiring models to include a core set of medical facts (FDA-approved medications, behavioral support, etc.) regardless of the user's persona, while still allowing the model to adapt its vocabulary and tone.
  4. Evaluation: 2,880 total responses were analyzed using automated regex-based metrics for word count, "hedging" phrases, redirect rates (telling the user to "talk to a doctor"), and fact coverage.
The twelve verbatim persona prefixes used in the study.

Datasets

The study utilized a popular-query set for smoking cessation derived from prior literature. The primary "dataset" produced and released by the authors consists of the 2,880 LLM responses generated during the audit, along with the analysis code. This serves as a benchmark for "identity-conditioned response consistency" in the health domain.

Key Insights and Ideas

  • The Expertise Penalty: In five of the six models, "Expert" personas (e.g., a pulmonologist) received fewer facts than novices. The models likely assumed the expert already knew the basics, leading them to omit crucial information. This creates a "reverse safety" issue where users are penalized for providing helpful context.
  • Age-Based Omission: At baseline, every model mentioned prescription medications to teens at substantially lower rates than to adults (a gap of 32 to 67 percentage points). While these drugs aren't first-line for minors, the models often remained silent rather than explaining the clinical nuance.
  • "Factual Floor" vs. Adaptation: The paper argues for a distinction between content and wrapping. A model should be free to use simpler language for a novice, but the underlying medical facts (the "floor") should remain constant if the clinical guidance is the same.
  • Prompt Non-Transferability: A system prompt that improved fact coverage in Claude and GPT actually caused Grok 4.3 to "over-compress" its answers, leading to a decrease in total facts. This suggests that safety or consistency prompts must be validated on a model-by-model basis.

Key Results

The intervention successfully leveled the playing field for most models:

  • Medication Mentions: Under the system prompt, the teen-to-adult difference in mentioning prescription medications fell to within 10 percentage points across all models.
  • Fact Coverage: In models like Claude Sonnet and GPT-5.5, the prompt significantly increased the total number of facts provided.
  • Redirects: The prompt reduced "authority-deflection" (telling the user to ask someone else instead of answering) across the board.
Figure 1: Aggregated key-fact mention rate for GPT-5.5 under both conditions. Rows are the twelve personas (life stage / expertise); columns are the seven fact categories. Under baseline (a), the Prescription Meds column is markedly pale for the three teen rows and the College/Novice row, contrasting with near-saturation for mid-career and retired rows; under the prompt (b), the column becomes uniformly dark while the Cold Turkey column lightens. Heatmaps for the other five models are in the Supplementary Materials.
Headline metrics across all six models and both conditions.

Limitations and Future Work

  • Confounded Personas: The personas varied by age and smoking history simultaneously (e.g., the teen was a vaper, the retiree a long-term smoker). Future work should isolate these variables.
  • Automated vs. Human Scoring: The study relied on word-matching. While efficient, it cannot distinguish between a model recommending a drug and warning against it. The authors emphasize that human clinical review is a necessary next step.
  • Single-Turn Bias: The audit only looked at the first response. Real-world health coaching is often multi-turn, where persona-driven biases might compound or dissipate over time.
  • Model Perishability: LLMs are updated constantly; these results represent a snapshot of model behavior from April–May 2026.

Summary updated

Abstract

Hundreds of millions of people now ask large language model (LLM) chatbots for health advice, which raises a question that is simple to pose and hard to answer: does everyone get the same advice? We audit six widely deployed LLMs from four providers on smoking cessation, a domain with codified clini- cal guidance and substantial public-health stakes, by putting the same four cessation questions to twelve user personas that differ in life stage, self-described expertise, and smoking history, for a total of 2,880 responses. What the models say depends on who appears to be asking. Every model named FDA-approved prescription cessation medications less often to teen personas than to adult personas, by 32 to 67 percentage points, and in five of six models a persona presenting health- care expertise received fewer facts than a persona presenting none. We then evaluate one deployable intervention: a short system prompt specifying a factual floor, the set of core med- ical facts every user should receive when they are relevant, while leaving tone, vocabulary, and framing free to adapt. The prompt narrows persona differences in every model, but its effect on overall fact coverage varies in both magnitude and direction across models, so a prompt validated on one system cannot be assumed to transfer to another. Two cautions frame these results. Equal mention rates are not always the correct clinical target, because prescription cessation pharmacother- apy is not recommended for minors, so a defensible factual floor must be age-aware rather than uniform. And because our personas vary on several attributes at once, we report scenario- level differences rather than isolated effects of age or expertise. We release all 2,880 responses and the analysis code.
BibTeX
@misc{yousefi2026same,
  author = {Artin Yousefi and Aizierjiang Aiersilan and Lorien C Abroms and David A Broniatowski},
  title = {{Same Question, Different Answers: Auditing Persona-Dependent Variation in Six LLMs on Smoking-Cessation Advice}},
  year = {2026},
  eprint = {2610.42039},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2610.42039},
}
Figure