TL;DR
This paper investigates whether Large Language Models (LLMs) exhibit a "superstar effect" by generating similar opinions about prominent figures across different languages. The study finds that LLMs tend to converge on a narrow set of globally recognized individuals, potentially limiting the representation of diverse cultural perspectives.
What Problem Does This Paper Address?
As LLMs become increasingly integrated into daily life for information retrieval and content generation, their outputs have the potential to significantly influence public opinion and shape global knowledge. While LLMs can converse in multiple languages, it's unclear if they reflect diverse cultural perspectives or if they homogenize opinions. This paper addresses the concern that LLMs might narrow the variety of perspectives by favoring globally recognized figures over locally important ones, a phenomenon known as the "superstar effect." This effect, observed in other technology-mediated domains, could lead to a homogenization of global knowledge representation.
Methods
The researchers designed an experimental setup with four dimensions: LLMs, languages, professional fields, and prompt adjectives. They used three prominent LLMs (GPT-4, Claude-3-Opus, Llama-3.1-70B-Instruct) and ten widely spoken languages. For each LLM and language, they generated responses to opinion-based prompts of the format "Who is the {adjective} {profession}?", varying fifteen professions and five descriptive adjectives. Prompts were translated into each target language, and responses were translated back to English for analysis. Named Entity Recognition (NER) was used to identify individuals mentioned. To measure consensus between language pairs, they used cosine similarity on response frequency vectors and compared it with lexical similarity data. The "superstar effect" was quantified using novelty scores and the Gini coefficient to measure inequality in name occurrences.
Datasets
The study did not introduce new datasets for training LLMs. Instead, it utilized existing LLMs and evaluated their responses. For cultural similarity analysis, they used the "Similarity Database of Modern Lexicons" by Bella et al. (2021).
Key Insights and Ideas
- The "Superstar Effect" is Real in LLMs: LLMs consistently favor a small number of highly recognized individuals across various professions and languages, demonstrating a clear superstar effect.
- Linguistic Diversity Doesn't Guarantee Opinion Diversity: Despite using multiple languages, LLMs often produce remarkably similar responses, suggesting that linguistic capabilities don't automatically translate to cultural nuance in opinion generation.
- Culture Matters, Especially Lexical Similarity: Languages with higher lexical similarity tend to yield more consensus in LLM responses, indicating that linguistic overlap can reflect deeper cultural interconnectedness that LLMs pick up on.
- Field-Specific Consensus: Scientific fields, with their universal principles and international collaborations, show higher consensus in LLM responses (e.g., Alan Turing for computer scientists). In contrast, fields like arts and politics, which are more culturally embedded, exhibit more diversity.
- LLM Choice Impacts Output: Different LLMs vary in their expansiveness (number of names) and novelty of responses, with Llama showing a tendency for more unique names.
Key Results
- On average, LLM responses contained 5.80 names, with a total of 2412 unique names identified across all responses.
- The Gini coefficient for name distribution across professions was consistently high (often above 0.70), indicating significant inequality and concentration on a few individuals. For example, Alan Turing appeared in 96.4% of responses for "computer scientist."
- Prompts in English tended to return the most names, while Arabic returned the fewest. Urdu showed a higher novelty score compared to other languages.
- Fields like 'poet' and 'writer,' which are more language-dependent, and subjective fields like 'military leader' and 'political figure,' led to more novel names in responses.
Limitations and Future Work
The authors acknowledge several limitations:
- Translation Dependency: All responses were translated back to English, which could introduce some bias, although they mitigated this by focusing on named entities.
- NER Bias: Named Entity Recognition might perform better for Western individuals than those from other cultures.
- Limited Scope: The study used only ten languages and three LLMs, and future work could explore less common languages or LLMs developed in different regions.
- Language as Proxy for Culture: Using language as a proxy for culture is a simplification, as significant differences exist within language groups.
- Model Updates: The use of current LLM versions means results might change with future updates, impacting reproducibility.
- Default Parameters: Using default LLM parameters reflects typical user interaction but doesn't explore how parameters like temperature might affect response diversity.
Future work suggestions include:
- Comparing with Human Responses: Investigating whether human opinions align with LLM outputs and if humans produce more or less diverse opinions.
- Culturally-Specific Prompts: Experimenting with prompts that explicitly ask for culturally-specific responses to see if this reduces opinion homogenization.
- Temporal Analysis: Evaluating how the superstar effect changes over time and its relationship with historical figures.
Summary updated