TL;DR
Large language models (LLMs) provide systematically different advice depending on a user's conversation history, but it isn't because they are secretly profiling user demographics. This paper reveals that LLMs actually struggle to infer traits like age or gender from context; instead, the topic of conversation acts as a subtle proxy, driving unexpected disparities in high-stakes advice.
What Problem Does This Paper Address?
When people use LLMs for high-stakes advice—such as legal, medical, or financial queries—previous studies noticed a troubling trend: even without explicitly stating your race, gender, or age, the model's advice shifts based on your prior conversation history. This often results in statistically significant disparities between sociodemographic groups (e.g., older users receiving more conservative political answers).
But why does this happen? Are LLMs internally profiling users based on subtle linguistic cues, or is something else pulling the strings? The authors investigate whether implicit demographic inference is truly the culprit behind these biased outcomes, or if other conversational features are driving the variance.
Methods
To untangle this, the authors took a multi-pronged approach across several open-weight models (Llama 3.1 8B, Gemma 3 12B, Qwen 3.6 27B) and a frontier reasoning model (Kimi K2.6):
- Disparity Measurement: The authors appended high-stakes advice questions to real conversation histories and measured the variance in model responses across demographic groups.
- Demographic Inference Testing: They evaluated if Kimi K2.6 could explicitly predict user demographics from text when prompted, and they trained linear probes on the internal latent representations of Gemma and Llama to see if demographic data is linearly encoded in the models' hidden states.
- Feature Importance Regression: They built ElasticNet and linear regression models using a massive array of features—including sociodemographics, psycholinguistic markers (LIWC), emotions, readability, and conversation topics—to see which features actually predict the model's final advice.
- Topic vs. Demographics (2x2 Design): They directly compared the influence of conversation topics against sociodemographics by analyzing pairs of users with the same/different demographics discussing the same/different topics.
Datasets
The study leverages three primary datasets:
- PRISM: A dataset of 8,011 real-world, multi-turn conversations with self-reported user demographics spanning 11 categories (age, ethnicity, religion, etc.) and clustered into 22 distinct topics.
- Community Alignment Dataset (CAD): 26,521 English conversations from users in India and the US, utilizing 560 pre-defined opening prompts as topics.
- Sociolinguistic Bias Benchmark (SBB): An evaluation set of high-stakes questions across five critical domains: government benefits, medical attention, legal issues, political views, and salary recommendations.
Key Insights and Ideas
- LLMs are Bad Demographic Detectors: Contrary to popular belief, LLMs struggle to guess a user's age, gender, or ethnicity from a single conversation history. Explicit prompting usually overpredicts the majority class, and internal linear probes show very low accuracy across model layers.
- Topics Rule the Roost: The actual subject matter of the conversation is the dominant force shaping the model's downstream answers. For instance, talking about luxury travel primes the model to suggest a higher starting salary for an unrelated job question, while talking about job hunting or budget constraints lowers it.
- The "Proxy" Trap: Even though models aren't directly profiling users, demographic bias still creeps in because topic choice is rarely neutral. Younger users are statistically more likely to discuss job searches, meaning the model's tendency to give lower salary recommendations to "job search" topics indirectly penalizes younger users. Topics act as persistent, implicit proxies for demographics.
Key Results
- Minimal Group Disparities: Group-level disparities are minimal in absolute terms. For example, gender-based salary recommendation differences were only around $100–$882, whereas differences between individual conversation histories swung by up to $20,000.
- Topic Dominance: Regression analysis confirmed that topic features consistently held the highest coefficient magnitudes across all domains and datasets.
- The 2x2 Analysis: As shown in the matrix below, two users from different demographic groups talking about the same topic received much closer answers than two users from the same demographic group talking about different topics.
Limitations and Future Work
The authors acknowledge several limitations:
- The study is limited to English conversations and heavily relies on US-centric sociodemographic categories.
- The datasets consist of crowdsourced interactions for AI alignment, which may not perfectly mirror organic, real-world user behavior.
- The artificial pairing of conversation histories with SBB questions might introduce unrealistic context shifts.
Finally, the authors experimented with a simple system prompt instructing the model to "be fair and unbiased." While it mitigated disparities in the political domain, it failed to fix the benefits or salary domains. The authors call for future work to develop more sophisticated, transparent mitigation techniques to handle these subtle, context-driven topic biases.
Summary updated