Assessing Historical Structural Oppression Worldwide via Rule-Guided Prompting of Large Language Models

Sreejato Chatterjee, Linh Tran, Quoc Duy Nguyen, Roni Kirson, Drue Hamlin, Harvest Aquino, Hanjia Lyu, Jiebo Luo, Timothy Dye

TL;DR

Measuring historical structural oppression globally is incredibly challenging due to highly localized histories and the rigid nature of traditional demographic categories. This paper introduces a novel framework that leverages Large Language Models (LLMs) combined with sociologically grounded, rule-guided prompting to transform free-text, self-identified ethnicity data into context-sensitive oppression scores. The approach achieves strong alignment with human experts, offering a scalable, cross-cultural tool for data-driven social science and public health research.

What Problem Does This Paper Address?

Traditional methods for quantifying structural disadvantage struggle with cross-national validity. Standardized census categories (like the US OMB standards) force individuals into broad, artificial buckets (e.g., "Asian" or "Hispanic") that flatten identity-based variations and mask massive internal heterogeneity.

On the other hand, material deprivation indices focus heavily on wealth, housing, or education. These metrics treat socioeconomic status as identity-neutral, frequently overlooking how race, ethnicity, and institutional power dynamics dictate access to those very resources. Because what constitutes a marginalized group varies drastically from one geopolitical context to another, a unified global framework has remained elusive. The authors address this gap by exploring whether the vast, cross-lingual knowledge embedded in LLMs can be harnessed to interpret open-ended, self-identified ethnic descriptions within their specific national histories.

Methods

The authors developed a five-level ordinal classification schema for identity-based historical oppression, ranging from Little to No Oppression (1) to Severe Oppression (5). To scale this schema using LLMs, they designed a Rule-Guided Prompting (RG) module.

Instead of letting the LLM freely guess (Vanilla prompting) or simply think step-by-step (Chain-of-Thought prompting), the RG module embeds six explicit sociological constraints directly into the prompt. These rules instruct the model to:

  1. Base classifications solely on historical and systemic factors (ignoring individual or purely economic success).
  2. Avoid assuming that globally marginalized identities experience oppression identically across all societies.
  3. Default national labels (e.g., "Canadian") to the dominant group unless modifiers are present.
  4. Lean toward the marginalized component in cases of mixed ancestry.
  5. Assign low scores if no explicit evidence of structural disadvantage exists.
  6. Reserve the highest scores (4 and 5) strictly for groups facing long-term, institutionalized exclusion across multiple major domains (housing, voting, education).

The researchers systematically evaluated these prompting strategies across three state-of-the-art models: Gemini 1.5 Pro, GPT-3.5 Turbo, and GPT-4o mini.

Fig. 1. Illustration of the prompting setup used for our experiment, comparing Rule-Guided Prompting (RG) with two baseline methods: Vanilla and Chain- of-Thought (CoT). Each prompt is constructed sequentially as follows: (1) system role →(2) identity statement →(3) instruction →(4) oppression schema →(5) optional CoT or RG instruction →(6) required output format. The complete set of rules for the RG method is provided in Section III-B2, and the set of guiding questions for the CoT method is detailed in Section IV-A1.

Datasets

The authors introduced HSO-Bench, an open-source benchmark dataset containing 334 free-text responses from a multilingual global COVID-19 impact study. The dataset spans 10 diverse countries: the United States, Canada, Australia, Brazil, Algeria, Palestine, Puerto Rico, Finland, Madagascar, and Sweden. To establish ground-truth historical oppression scores, human experts trained in sociological methods annotated each response, providing detailed rationales and academic citations for their classifications.

Key Insights and Ideas

  • Unguided Chain-of-Thought (CoT) Can Backfire: Surprisingly, unguided CoT prompting led to worse performance and a higher rate of overestimation errors than vanilla prompting. Without explicit sociological guardrails, LLMs tend to over-intellectualize, overgeneralize broad narratives, and hallucinate historical grievances.
  • Geographic Bias in LLM Knowledge: LLMs aligned closely with human experts in Western nations (the US, Canada, Australia) and Brazil, where sociopolitical narratives are heavily documented in the models' training data. However, alignment dropped significantly in non-Western contexts like Algeria and Madagascar.
  • The "Colonial Conflation" Error: In Algeria, LLMs frequently misclassified the dominant national majority ("Arabo-musulman") as highly oppressed. The models over-indexed on the historical legacy of French colonization, failing to recognize that this group currently holds institutional power.

Key Results

  • Gemini 1.5 Pro using the rule-guided prompting strategy was the top performer, achieving a strong Pearson correlation coefficient of 0.852 with human expert annotations, a Mean Absolute Error (MAE) of 0.401, and the highest classification accuracy (0.608).
  • GPT-4o mini delivered competitive baseline results in vanilla mode but failed to improve significantly under rule-guided constraints, suggesting its smaller architecture might limit its ability to process multi-layered sociological rules.
  • Error Reduction: Across all tested models, the rule-guided prompting module drastically minimized high-magnitude errors (where the model's score deviated from the human score by 2 or more levels).
Model performance metrics across different prompting strategies.

Limitations and Future Work

The current framework focuses exclusively on ethnicity and race, omitting other critical, intersecting axes of identity such as gender, sexual orientation, and disability. Additionally, the benchmark dataset is relatively small ($n=334$), meaning certain countries have limited representation, and the authors did not calculate standard intercoder reliability metrics (like Kappa) for the initial human ground truth.

To build on this foundation, the authors suggest that future work should incorporate Retrieval-Augmented Generation (RAG) or few-shot calibration. Dynamically injecting localized, country-specific historical context into the LLM's context window could significantly reduce bias and improve accuracy in underperforming, non-Western regions.

Summary updated

Abstract

Traditional efforts to measure historical structural oppression struggle with cross-national validity due to the unique, locally specified histories of exclusion, colonization, and social status in each country, and often have relied on structured indices that privilege material resources while overlooking lived, identity-based exclusion. We introduce a novel framework for oppression measurement that leverages Large Language Models (LLMs) to generate context-sensitive scores of lived historical disadvantage across diverse geopolitical settings. Using unstructured self-identified ethnicity utterances from a multilingual COVID-19 global study, we design rule-guided prompting strategies that encourage models to produce interpretable, theoretically grounded estimations of oppression. We systematically evaluate these strategies across multiple state-of-the-art LLMs. Our results demonstrate that LLMs, when guided by explicit rules, can capture nuanced forms of identity-based historical oppression within nations. This approach provides a complementary measurement tool that highlights dimensions of systemic exclusion, offering a scalable, cross-cultural lens for understanding how oppression manifests in data-driven research and public health contexts. To support reproducible evaluation, we release an open-sourced benchmark dataset for assessing LLMs on oppression measurement (https://github.com/chattergpt/HSO-Bench).
BibTeX
@misc{chatterjee2025assessing,
  author = {Sreejato Chatterjee and Linh Tran and Quoc Duy Nguyen and Roni Kirson and Drue Hamlin and Harvest Aquino and Hanjia Lyu and Jiebo Luo and Timothy Dye},
  title = {{Assessing Historical Structural Oppression Worldwide via Rule-Guided Prompting of Large Language Models}},
  year = {2025},
  eprint = {2509.15216},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2509.15216},
}
Figure