TL;DR
Large Language Models (LLMs) like GPT-40, Claude 3.5 Sonnet, and Gemini 1.5 Pro exhibit significant biases in recognizing scientists, mirroring and sometimes amplifying existing disparities in scientific recognition based on gender and geography. This study introduces a novel framework to audit LLM recognition, revealing that while LLMs can be powerful tools for accessing scientific knowledge, their inherent biases necessitate careful consideration and development of more inclusive AI systems.
What Problem Does This Paper Address?
As Large Language Models (LLMs) become increasingly integrated into scientific workflows for information retrieval, summarization, and even discovery, a critical question arises: how well do these models understand and represent the scientific community itself? Specifically, can LLMs accurately recognize individual scientists, a fundamental aspect of understanding the social structure and expertise distribution within science? This paper addresses the problem of unequal scientific recognition within LLMs, investigating whether these models perpetuate or even exacerbate existing biases related to gender, geography, and scholarly impact. The authors aim to provide a systematic evaluation of LLM recognition capabilities, using authoritative knowledge bases as a baseline to understand how LLMs reflect and potentially amplify societal inequalities in science.
Methods
The study employs a multi-stage evaluation pipeline to audit scientific recognition in LLMs. This framework combines:
- Profile-Level Prompting: A structured prompt (Figure 2) was designed to query LLMs for detailed profiles of 100,000 physicists, including research areas, affiliations, key collaborators, and major contributions. Responses were fact-checked for accuracy.
- Mutual Recognition Analysis: This involves assessing the consistency of recognition across different LLMs using Venn diagrams and Cohen's Kappa (κ) to quantify agreement. This goes beyond simple coverage metrics to understand how models overlap in their recognition of entities.
- Bias Auditing: The study computes the marginal effect of group membership (e.g., gender, region) on recognition probability to identify systematic biases. This is done using logistic regression, controlling for factors like scholarly impact (citations, productivity) and career stage.
- Data Provenance Analysis: To investigate the role of training data, the authors use both observational analysis (comparing recognition with and without Wikipedia profiles) and a natural experiment based on training cutoffs for Claude.
Datasets
- OpenAlex: A scholarly database containing over 250 million works, used to curate a sample of 100,000 physicists for evaluation.
- Wikipedia: A large, volunteer-curated knowledge repository with over 2.29 million individual profiles, used as a baseline for comparison and for analyzing data provenance.
- DeepSeek-V3: An open-weight model used in certain experiments to show that observed patterns hold across different model architectures.
No new benchmarks were introduced; instead, the study leverages existing authoritative sources (OpenAlex, Wikipedia) to evaluate LLM performance.
Key Insights and Ideas
- LLMs Amplify Existing Biases: The most striking insight is that LLMs, despite their advanced capabilities, not only reflect but often amplify existing disparities in scientific recognition, particularly concerning gender and geography.
- Wikipedia as a Baseline for Parity: Interestingly, Wikipedia, despite its own potential biases, demonstrates a more equitable recognition pattern for gender compared to proprietary LLMs, suggesting that LLMs are actively amplifying these biases.
- Impact of Data Provenance is Crucial: The study provides causal evidence that exposure to public knowledge sources like Wikipedia significantly boosts a scientist's recognition by LLMs, highlighting the critical role of training data in shaping these outcomes.
- High-Impact Scientists Receive More Consistent Recognition: LLMs show stronger agreement in recognizing high-impact scientists (those with >10,000 citations), indicating a convergence towards academic prominence, which itself can be a source of bias.
Key Results
- Unequal Recognition: LLMs exhibit selective and inconsistent recognition patterns. Recognition strongly correlates with scholarly impact (citations), but remains uneven across gender and geography.
- Gender Disparities: All proprietary LLMs show higher recognition for men. Even after controlling for scientific impact and career stage, significant gender disparities persist, with women researchers being underrecognized. This pattern holds even in fields with higher female representation.
- Geographic Disparities: Researchers affiliated with institutions in North America, Oceania, and Western Europe are significantly more likely to be recognized than those in Asia, South America, and Africa, even after controlling for impact and career stage.
- Wikipedia's Influence: Scientists with Wikipedia profiles are significantly more likely to be recognized by all LLMs. A natural experiment showed a 27% increase in recognition probability due to Wikipedia exposure.
- Model Performance: Claude generally shows higher overall coverage, followed by Gemini and GPT. Agreement in recognition is higher among high-impact scientists.
Limitations and Future Work
- Focus on Physicists: The study's primary focus on physicists, a historically male-dominated field, might influence observed gender disparities. Supplementary experiments in other fields were conducted, but a broader cross-disciplinary evaluation is suggested.
- Baseline Limitations: Treating OpenAlex and Wikipedia as baselines has inherent limitations, as these sources may also contain biases. Future work could extend baselines to pre-training corpora to directly link corpus exposure to recognition behavior.
- Observational vs. Internal Biases: The study focuses on observational representation gaps rather than internal model mechanisms. Future research could explore latent factors driving biases and investigate model internals for deeper interpretability.
- Future Directions: The authors suggest extending analyses to publicly documented pre-training corpora, exploring model internals for interpretability, and developing more transparent and inclusive knowledge systems. They also propose further work on distinguishing "missing knowledge" from representation geometry or decision-time heuristics.
Summary updated