TL;DR
Measuring historical structural oppression globally is incredibly challenging due to highly localized histories and the rigid nature of traditional demographic categories. This paper introduces a novel framework that leverages Large Language Models (LLMs) combined with sociologically grounded, rule-guided prompting to transform free-text, self-identified ethnicity data into context-sensitive oppression scores. The approach achieves strong alignment with human experts, offering a scalable, cross-cultural tool for data-driven social science and public health research.
What Problem Does This Paper Address?
Traditional methods for quantifying structural disadvantage struggle with cross-national validity. Standardized census categories (like the US OMB standards) force individuals into broad, artificial buckets (e.g., "Asian" or "Hispanic") that flatten identity-based variations and mask massive internal heterogeneity.
On the other hand, material deprivation indices focus heavily on wealth, housing, or education. These metrics treat socioeconomic status as identity-neutral, frequently overlooking how race, ethnicity, and institutional power dynamics dictate access to those very resources. Because what constitutes a marginalized group varies drastically from one geopolitical context to another, a unified global framework has remained elusive. The authors address this gap by exploring whether the vast, cross-lingual knowledge embedded in LLMs can be harnessed to interpret open-ended, self-identified ethnic descriptions within their specific national histories.
Methods
The authors developed a five-level ordinal classification schema for identity-based historical oppression, ranging from Little to No Oppression (1) to Severe Oppression (5). To scale this schema using LLMs, they designed a Rule-Guided Prompting (RG) module.
Instead of letting the LLM freely guess (Vanilla prompting) or simply think step-by-step (Chain-of-Thought prompting), the RG module embeds six explicit sociological constraints directly into the prompt. These rules instruct the model to:
- Base classifications solely on historical and systemic factors (ignoring individual or purely economic success).
- Avoid assuming that globally marginalized identities experience oppression identically across all societies.
- Default national labels (e.g., "Canadian") to the dominant group unless modifiers are present.
- Lean toward the marginalized component in cases of mixed ancestry.
- Assign low scores if no explicit evidence of structural disadvantage exists.
- Reserve the highest scores (4 and 5) strictly for groups facing long-term, institutionalized exclusion across multiple major domains (housing, voting, education).
The researchers systematically evaluated these prompting strategies across three state-of-the-art models: Gemini 1.5 Pro, GPT-3.5 Turbo, and GPT-4o mini.
Datasets
The authors introduced HSO-Bench, an open-source benchmark dataset containing 334 free-text responses from a multilingual global COVID-19 impact study. The dataset spans 10 diverse countries: the United States, Canada, Australia, Brazil, Algeria, Palestine, Puerto Rico, Finland, Madagascar, and Sweden. To establish ground-truth historical oppression scores, human experts trained in sociological methods annotated each response, providing detailed rationales and academic citations for their classifications.
Key Insights and Ideas
- Unguided Chain-of-Thought (CoT) Can Backfire: Surprisingly, unguided CoT prompting led to worse performance and a higher rate of overestimation errors than vanilla prompting. Without explicit sociological guardrails, LLMs tend to over-intellectualize, overgeneralize broad narratives, and hallucinate historical grievances.
- Geographic Bias in LLM Knowledge: LLMs aligned closely with human experts in Western nations (the US, Canada, Australia) and Brazil, where sociopolitical narratives are heavily documented in the models' training data. However, alignment dropped significantly in non-Western contexts like Algeria and Madagascar.
- The "Colonial Conflation" Error: In Algeria, LLMs frequently misclassified the dominant national majority ("Arabo-musulman") as highly oppressed. The models over-indexed on the historical legacy of French colonization, failing to recognize that this group currently holds institutional power.
Key Results
- Gemini 1.5 Pro using the rule-guided prompting strategy was the top performer, achieving a strong Pearson correlation coefficient of 0.852 with human expert annotations, a Mean Absolute Error (MAE) of 0.401, and the highest classification accuracy (0.608).
- GPT-4o mini delivered competitive baseline results in vanilla mode but failed to improve significantly under rule-guided constraints, suggesting its smaller architecture might limit its ability to process multi-layered sociological rules.
- Error Reduction: Across all tested models, the rule-guided prompting module drastically minimized high-magnitude errors (where the model's score deviated from the human score by 2 or more levels).
Limitations and Future Work
The current framework focuses exclusively on ethnicity and race, omitting other critical, intersecting axes of identity such as gender, sexual orientation, and disability. Additionally, the benchmark dataset is relatively small ($n=334$), meaning certain countries have limited representation, and the authors did not calculate standard intercoder reliability metrics (like Kappa) for the initial human ground truth.
To build on this foundation, the authors suggest that future work should incorporate Retrieval-Augmented Generation (RAG) or few-shot calibration. Dynamically injecting localized, country-specific historical context into the LLM's context window could significantly reduce bias and improve accuracy in underperforming, non-Western regions.
Summary updated