Topics as Proxies for Sociodemographics: How Conversational Context Affects LLM Answers

Vera Neplenbroek, Gabriele Sarti, Arianna Bisazza, Raquel FernándezarXiv 2026 · arXiv

TL;DR

Large language models (LLMs) provide systematically different advice depending on a user's conversation history, but it isn't because they are secretly profiling user demographics. This paper reveals that LLMs actually struggle to infer traits like age or gender from context; instead, the topic of conversation acts as a subtle proxy, driving unexpected disparities in high-stakes advice.

What Problem Does This Paper Address?

When people use LLMs for high-stakes advice—such as legal, medical, or financial queries—previous studies noticed a troubling trend: even without explicitly stating your race, gender, or age, the model's advice shifts based on your prior conversation history. This often results in statistically significant disparities between sociodemographic groups (e.g., older users receiving more conservative political answers).

But why does this happen? Are LLMs internally profiling users based on subtle linguistic cues, or is something else pulling the strings? The authors investigate whether implicit demographic inference is truly the culprit behind these biased outcomes, or if other conversational features are driving the variance.

Methods

To untangle this, the authors took a multi-pronged approach across several open-weight models (Llama 3.1 8B, Gemma 3 12B, Qwen 3.6 27B) and a frontier reasoning model (Kimi K2.6):

  1. Disparity Measurement: The authors appended high-stakes advice questions to real conversation histories and measured the variance in model responses across demographic groups.
  2. Demographic Inference Testing: They evaluated if Kimi K2.6 could explicitly predict user demographics from text when prompted, and they trained linear probes on the internal latent representations of Gemma and Llama to see if demographic data is linearly encoded in the models' hidden states.
  3. Feature Importance Regression: They built ElasticNet and linear regression models using a massive array of features—including sociodemographics, psycholinguistic markers (LIWC), emotions, readability, and conversation topics—to see which features actually predict the model's final advice.
  4. Topic vs. Demographics (2x2 Design): They directly compared the influence of conversation topics against sociodemographics by analyzing pairs of users with the same/different demographics discussing the same/different topics.

Datasets

The study leverages three primary datasets:

  • PRISM: A dataset of 8,011 real-world, multi-turn conversations with self-reported user demographics spanning 11 categories (age, ethnicity, religion, etc.) and clustered into 22 distinct topics.
  • Community Alignment Dataset (CAD): 26,521 English conversations from users in India and the US, utilizing 560 pre-defined opening prompts as topics.
  • Sociolinguistic Bias Benchmark (SBB): An evaluation set of high-stakes questions across five critical domains: government benefits, medical attention, legal issues, political views, and salary recommendations.

Key Insights and Ideas

  • LLMs are Bad Demographic Detectors: Contrary to popular belief, LLMs struggle to guess a user's age, gender, or ethnicity from a single conversation history. Explicit prompting usually overpredicts the majority class, and internal linear probes show very low accuracy across model layers.
  • Topics Rule the Roost: The actual subject matter of the conversation is the dominant force shaping the model's downstream answers. For instance, talking about luxury travel primes the model to suggest a higher starting salary for an unrelated job question, while talking about job hunting or budget constraints lowers it.
  • The "Proxy" Trap: Even though models aren't directly profiling users, demographic bias still creeps in because topic choice is rarely neutral. Younger users are statistically more likely to discuss job searches, meaning the model's tendency to give lower salary recommendations to "job search" topics indirectly penalizes younger users. Topics act as persistent, implicit proxies for demographics.

Key Results

  • Minimal Group Disparities: Group-level disparities are minimal in absolute terms. For example, gender-based salary recommendation differences were only around $100–$882, whereas differences between individual conversation histories swung by up to $20,000.
Figure 2: Significant differences in each model’s av- erage recommended salary across gender groups in PRISM. The horizontal dashed line is the model’s baseline prediction without any conversational history. While differences across gender groups are statistically significant for all models (p < 0.01, indicated by *), they are in the order of magnitude of $100, and smaller than differences with respect to the model’s baseline.
  • Topic Dominance: Regression analysis confirmed that topic features consistently held the highest coefficient magnitudes across all domains and datasets.
  • The 2x2 Analysis: As shown in the matrix below, two users from different demographic groups talking about the same topic received much closer answers than two users from the same demographic group talking about different topics.
Average difference in Gemma's predictions between users based on topic vs. demographic group.

Limitations and Future Work

The authors acknowledge several limitations:

  • The study is limited to English conversations and heavily relies on US-centric sociodemographic categories.
  • The datasets consist of crowdsourced interactions for AI alignment, which may not perfectly mirror organic, real-world user behavior.
  • The artificial pairing of conversation histories with SBB questions might introduce unrealistic context shifts.

Finally, the authors experimented with a simple system prompt instructing the model to "be fair and unbiased." While it mitigated disparities in the political domain, it failed to fix the benefits or salary domains. The authors call for future work to develop more sophisticated, transparent mitigation techniques to handle these subtle, context-driven topic biases.

Summary updated

Abstract

When large language models (LLMs) are used in high-stakes scenarios, such as legal, medical and financial advice, even a single conversation history is enough to drive differences in outcomes between users. Prior work has demonstrated that this results in outcome disparities between sociodemographic groups, with some groups receiving more advantageous outcomes than others. In this work, we demonstrate that LLMs actually struggle to infer user sociodemographics from a single conversation history and that although there are disparities between sociodemographic groups, they are minimal in magnitude. To investigate what is the main driver of disparities between users, we compare user sociodemographics to a range of (psycho)linguistic features of conversations, including conversation topic, emotions, and readability. We find that conversation topics are most predictive of LLM-generated advice within a conversational context, which, to some extent, function as proxies for sociodemographic groups and often affect advice in unpredictable ways. This is cause for concern and highlights the need for future research to better understand the effect of conversational context on LLM outputs in high-stakes scenarios.
BibTeX
@misc{neplenbroek2026topics,
  author = {Vera Neplenbroek and Gabriele Sarti and Arianna Bisazza and Raquel Fernández},
  title = {{Topics as Proxies for Sociodemographics: How Conversational Context Affects LLM Answers}},
  year = {2026},
  eprint = {2606.02776},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2606.02776},
}
Figure