In the sensitive domain of sexual and reproductive health (SRH), context is paramount. A medical recommendation for a teenager in a state with restrictive healthcare laws may differ significantly from advice given to an older adult in a more liberal jurisdiction. However, users often pose "underspecified" questions to Large Language Models (LLMs)—queries that lack the necessary personal context for a precise answer.
Presented at EMNLP 2024, this research investigates how LLMs handle these vague queries. The study reveals that when models are forced to provide a single answer to an underspecified question, they exhibit significant demographic biases, consistently defaulting to answers tailored for a specific "typical" user.
Methodology: Probing for Hidden Defaults
To uncover these biases, researchers employed a "probing" methodology. They first asked models a question without context (the "original" answer) and then asked the same question while providing specific demographic attributes (e.g., "I am a 25-year-old male in Wyoming").
By calculating the cosine similarity between the original answer and various contextual answers using SentenceBERT embeddings, the team could quantify which demographics the model "defaults" to. They also tracked a "Percent Win" metric—how often a specific group's contextual answer was the most similar to the default—and conducted human evaluations with healthcare professionals to assess the appropriateness of the answers.
The CONTEXTSRH Dataset
The study introduced CONTEXTSRH, a new dataset of 187 U.S.-based English contextual questions. These questions were sourced from Planned Parenthood and Columbia University’s "Go Ask Alice" blogs. Each question was labeled based on its dependency on three key attributes:
- Age: 10 milestone ages ranging from 10 to 70.
- Biological Sex: Male and Female.
- Location: 6 U.S. states representing a spectrum of political and legal environments regarding reproductive health.
Key Results and Demographic Bias
The evaluation of models like GPT-3.5-turbo, LLaMA-2-70b-chat, and LLaMA-3-70b-chat revealed a clear mental model of the "typical" SRH information seeker: a woman in her 20s living in a liberal state.
Age and Sex Bias
Across all models, the highest similarity scores were found for the 18–30 age bracket. For instance, in GPT-3.5-turbo, the "Percent Win" for age 18 was 20.7%, while for age 70, it was only 5.7%. Similarly, default answers aligned with the Female context 60.9% of the time in GPT-3.5, compared to only 39.1% for Male. This often led to "male erasure," where models focused exclusively on vaginal symptoms for conditions like yeast infections unless explicitly prompted otherwise.
Geographic and Legal Bias
When location was a factor, models' default answers aligned most closely with the laws of Massachusetts (MA), which had a 40% Win rate in LLaMA-2. Conversely, conservative states like South Dakota or Wyoming had significantly lower alignment. Furthermore, models often struggled with the rapidly changing legal landscape post-Roe v. Wade, providing generic or outdated advice that could be medically or legally risky for users in restrictive states.
Moving Toward Clarification-Aware Models
The study highlights that these hidden defaults can lead to knowledge conflicts and health disparities. While human evaluators found default answers were correct for females 98.4% of the time, that accuracy dropped to 83.9% for males, illustrating the real-world impact of these biases.
To mitigate these risks, the authors suggest that for high-stakes domains like health, models should be integrated with Retrieval-Augmented Generation (RAG) to access up-to-date information. More importantly, they call for models to become "clarification-aware": rather than guessing a demographic default, the model should recognize the underspecification and ask the user for the necessary context before providing a medical recommendation.
Summary updated