Abstract
Large language models (LLMs) are trained on vast amounts of data to generate natural language, enabling them to perform tasks like text summarization and question answering. These models have become popular in artificial intelligence (AI) assistants like ChatGPT and already play an influential role in how humans access information. However, the behavior of LLMs varies depending on their design, training, and use. In this paper, we uncover notable diversity in the ideological stance exhibited across different LLMs and languages in which they are accessed. We do this by prompting a diverse panel of popular LLMs to describe a large number of prominent and controversial personalities from recent world history, both in English and in Chinese. By identifying and analyzing moral assessments reflected in the generated descriptions, we find consistent normative differences between how the same LLM responds in Chinese compared to English. Similarly, we identify normative disagreements between Western and non-Western LLMs about prominent actors in geopolitical conflicts. Furthermore, popularly hypothesized disparities in political goals among Western models are reflected in significant normative differences related to inclusion, social inequality, and political scandals. Our results show that the ideological stance of an LLM often reflects the worldview of its creators. This raises important concerns around technological and regulatory efforts with the stated aim of making LLMs ideologically 'unbiased', and it poses risks for political instrumentalization.
Introduction
As LLMs increasingly serve as the primary gatekeepers of digital information—powering search engines, chatbots, and writing assistants—the question of their "trustworthiness" has moved to the forefront of AI research. While significant attention has been paid to factuality and the prevention of "hallucinations," the subtler issue of embedded ideology remains difficult to quantify. Previous attempts to measure bias often relied on direct questionnaires, which models can be fine-tuned to answer with artificial neutrality. This study seeks to elicit the implicit ideology of LLMs through open-ended descriptions of political figures, providing a more ecologically valid measure of how these models behave during natural interactions.
Methodology
The researchers employed a two-stage prompting strategy designed to bypass hard-coded neutrality guardrails. In the first stage, 19 prominent LLMs (including GPT-4o, Claude 3.5, Gemini, and Llama 3.1) were asked to describe 3,991 political figures from the Pantheon dataset in six different languages. In the second stage, the models were shown their own descriptions in a fresh context and asked to rate the moral sentiment on a five-point Likert scale.
To interpret these assessments, political figures were mapped to 61 ideological tags adapted from the Manifesto Project (e.g., "Environmentalism," "Marxism," "Free Market"). Principal Component Analysis (PCA) was then used to visualize the ideological spectrum across different model-language pairs.
Results and Discussion
Geopolitical and Linguistic Clustering
The study found that LLMs are not neutral information brokers; rather, they reflect the political and moral values of their creators and the cultures associated with the languages they process. PCA revealed that the first principal component, accounting for 54.7% of the variance, clearly separates "progressive pluralism" (typical of Western models and languages) from "conservative nationalism" (more prevalent in non-Western contexts).
Models clustered strongly by region of origin. Western models generally leaned toward inclusivity and sustainability, while models from China, Russia, and the UAE reflected nationalistic or region-specific priorities. Notably, the same model often exhibited different values depending on the language of the prompt. For instance, Chinese-prompted responses were generally more critical of "Constitutional Reform" and more positive toward "Supply-side Economics" than their English counterparts.
Divergent Perspectives on Political Figures
The ideological divide is most apparent when comparing how models from different regions rate specific historical and contemporary figures. There is a clear normative disagreement between Western and non-Western LLMs regarding actors in geopolitical conflicts.
Within the Chinese AI landscape, a split was identified: models like Alibaba’s Qwen appeared more internationally focused, while Baidu’s Wenxiaoyan aligned more closely with domestic policy values. Similarly, in the US, Google’s Gemini emerged as the most "progressive" regarding social equality, while xAI’s Grok leaned more toward national sovereignty.
Conclusion
The findings suggest that "ideological neutrality" is a culturally defined concept and likely impossible to achieve in practice. The ideological stance of an LLM is an inherent reflection of its training data and the worldview of its creators. This reality poses challenges for regulatory efforts aimed at making AI "unbiased" and highlights the potential for the political instrumentalization of these powerful tools. Future work may involve developing public dashboards to help users navigate the ideological "personalities" of the models they employ.
Summary updated