Introduction
Measuring social inequality on a global scale presents a significant challenge for researchers. Traditional economic metrics, such as the Index of Multiple Deprivation (IMD), are often limited to material resources like income and housing and are typically confined to specific national contexts. Furthermore, standardized census categories—such as "Asian" or "Hispanic"—are frequently too broad to capture the localized histories of exclusion and colonization that define lived experiences. A group that is marginalized in one nation may be dominant in another, necessitating a measurement framework sensitive to local historical nuances.
To address this, we present a novel framework that leverages Large Language Models (LLMs) to quantify historical structural oppression. By employing "rule-guided prompting," we transform LLMs into "synthetic sociologists" capable of processing unstructured, free-text identity descriptions with high accuracy.
Methodology: Rule-Guided Prompting
The core of this research is the Rule-Guided Prompting (RG) framework. Unlike standard "Vanilla" prompting or "Chain-of-Thought" (CoT) reasoning, RG embeds specific sociological constraints directly into the model's instructions to prevent common inferential errors.
The framework utilizes a five-level ordinal scale for oppression, ranging from "Little to No Oppression" (privileged groups) to "Severe Oppression" (groups facing systemic violence). Six specific rules guide the model's judgment:
- Contextual Sensitivity: Models are instructed not to assume uniform oppression levels for a group across different countries.
- Default Assumptions: National labels (e.g., "Canadian") are assumed to refer to the dominant group unless specified otherwise.
- Mixed Ancestry: Following historical "one-drop" logic, the model is directed to lean toward the rating of the more historically marginalized component of a mixed identity.
Experimental Setup and Datasets
The framework was evaluated using HSO-Bench (Historical Structural Oppression Benchmark), a new dataset comprising 334 free-text responses from a multilingual global COVID-19 study. These responses represent 10 diverse regions, including the US, Brazil, Algeria, Palestine, and Madagascar. Human expert "ground truth" scores were used to evaluate the performance of three models: Gemini 1.5 Pro, GPT-3.5 Turbo, and GPT-4o mini.
Results and Geographic Disparities
The results demonstrate that LLMs, when properly constrained, can closely approximate human sociological judgment. Gemini 1.5 Pro using Rule-Guided prompting emerged as the top performer, achieving a Pearson correlation of 0.852 and a Mean Absolute Error (MAE) of 0.401. In contrast, GPT-3.5 Turbo consistently underperformed, and while GPT-4o mini showed promise in baseline tasks, it struggled to integrate complex sociological rules as effectively as Gemini.
A key finding was the significant geographic variation in model performance. Alignment was highest in Western contexts (Brazil, Canada, US), where sociopolitical narratives are extensively documented in training data. However, performance dipped significantly in regions like Algeria ($r=0.52$) and Madagascar ($r=0.50$), where models struggled with localized ethnic nuances and domain-specific histories.
Discussion and Insights
Several critical insights emerged from the error analysis:
- Overestimation Bias: Models tended to overestimate the severity of oppression, often projecting broad colonial narratives onto national majorities.
- CoT Limitations: Chain-of-Thought prompting occasionally decreased performance. Without explicit rules, the models' internal reasoning often led to incorrect historical assumptions, such as misidentifying "Indian" in a US context.
- The Power of Constraints: Explicitly encoding sociological rules was the most effective method for reducing error and mitigating the impact of global stereotypes.
Conclusion and Future Work
While this framework offers a scalable method for measuring identity-based disadvantage, limitations remain. The current models exhibit a clear geographic bias toward Western history. Future iterations could incorporate Retrieval-Augmented Generation (RAG) to provide models with specific local historical contexts for underrepresented regions. Additionally, expanding the framework to include intersectional factors such as gender, disability, and sexual orientation is a vital next step. By releasing HSO-Bench, we aim to foster the development of more culturally nuanced AI systems capable of understanding the deep roots of social exclusion worldwide.
Summary updated