Assessing Historical Structural Oppression Worldwide via Rule-Guided Prompting of Large Language Models

Sanjay Chatterjee, Linh Tran, Quoc Duy Nguyen, Roni Kirson, Drue Hamlin, Harvest Aquino, Hanjia Lyu, Jiebo Luo, Timothy Dye

Introduction

Measuring social inequality on a global scale presents a significant challenge for researchers. Traditional economic metrics, such as the Index of Multiple Deprivation (IMD), are often limited to material resources like income and housing and are typically confined to specific national contexts. Furthermore, standardized census categories—such as "Asian" or "Hispanic"—are frequently too broad to capture the localized histories of exclusion and colonization that define lived experiences. A group that is marginalized in one nation may be dominant in another, necessitating a measurement framework sensitive to local historical nuances.

To address this, we present a novel framework that leverages Large Language Models (LLMs) to quantify historical structural oppression. By employing "rule-guided prompting," we transform LLMs into "synthetic sociologists" capable of processing unstructured, free-text identity descriptions with high accuracy.

Methodology: Rule-Guided Prompting

The core of this research is the Rule-Guided Prompting (RG) framework. Unlike standard "Vanilla" prompting or "Chain-of-Thought" (CoT) reasoning, RG embeds specific sociological constraints directly into the model's instructions to prevent common inferential errors.

Comparison of Vanilla, Chain-of-Thought, and Rule-Guided prompting strategies.

The framework utilizes a five-level ordinal scale for oppression, ranging from "Little to No Oppression" (privileged groups) to "Severe Oppression" (groups facing systemic violence). Six specific rules guide the model's judgment:

  • Contextual Sensitivity: Models are instructed not to assume uniform oppression levels for a group across different countries.
  • Default Assumptions: National labels (e.g., "Canadian") are assumed to refer to the dominant group unless specified otherwise.
  • Mixed Ancestry: Following historical "one-drop" logic, the model is directed to lean toward the rating of the more historically marginalized component of a mixed identity.

Experimental Setup and Datasets

The framework was evaluated using HSO-Bench (Historical Structural Oppression Benchmark), a new dataset comprising 334 free-text responses from a multilingual global COVID-19 study. These responses represent 10 diverse regions, including the US, Brazil, Algeria, Palestine, and Madagascar. Human expert "ground truth" scores were used to evaluate the performance of three models: Gemini 1.5 Pro, GPT-3.5 Turbo, and GPT-4o mini.

Results and Geographic Disparities

The results demonstrate that LLMs, when properly constrained, can closely approximate human sociological judgment. Gemini 1.5 Pro using Rule-Guided prompting emerged as the top performer, achieving a Pearson correlation of 0.852 and a Mean Absolute Error (MAE) of 0.401. In contrast, GPT-3.5 Turbo consistently underperformed, and while GPT-4o mini showed promise in baseline tasks, it struggled to integrate complex sociological rules as effectively as Gemini.

Mean correlation (r) between LLM and human expert ratings across different countries.

A key finding was the significant geographic variation in model performance. Alignment was highest in Western contexts (Brazil, Canada, US), where sociopolitical narratives are extensively documented in training data. However, performance dipped significantly in regions like Algeria ($r=0.52$) and Madagascar ($r=0.50$), where models struggled with localized ethnic nuances and domain-specific histories.

Discussion and Insights

Several critical insights emerged from the error analysis:

  1. Overestimation Bias: Models tended to overestimate the severity of oppression, often projecting broad colonial narratives onto national majorities.
  2. CoT Limitations: Chain-of-Thought prompting occasionally decreased performance. Without explicit rules, the models' internal reasoning often led to incorrect historical assumptions, such as misidentifying "Indian" in a US context.
  3. The Power of Constraints: Explicitly encoding sociological rules was the most effective method for reducing error and mitigating the impact of global stereotypes.

Conclusion and Future Work

While this framework offers a scalable method for measuring identity-based disadvantage, limitations remain. The current models exhibit a clear geographic bias toward Western history. Future iterations could incorporate Retrieval-Augmented Generation (RAG) to provide models with specific local historical contexts for underrepresented regions. Additionally, expanding the framework to include intersectional factors such as gender, disability, and sexual orientation is a vital next step. By releasing HSO-Bench, we aim to foster the development of more culturally nuanced AI systems capable of understanding the deep roots of social exclusion worldwide.

Summary updated

Abstract

Traditional efforts to measure historical structural oppression struggle with cross-national validity due to the unique, locally specified histories of exclusion, colonization, and social status in each country, and often have relied on structured indices that privilege material resources while overlooking lived, identity-based exclusion. We introduce a novel framework for oppression measurement that leverages Large Language Models (LLMs) to generate context-sensitive scores of lived historical disadvantage across diverse geopolitical settings. Using unstructured self-identified ethnicity utterances from a multilingual COVID-19 global study, we design rule-guided prompting strategies that encourage models to produce interpretable, theoretically grounded estimations of oppression. We systematically evaluate these strategies across multiple state-of-the-art LLMs. Our results demonstrate that LLMs, when guided by explicit rules, can capture nuanced forms of identity-based historical oppression within nations. This approach provides a complementary measurement tool that highlights dimensions of systemic exclusion, offering a scalable, cross-cultural lens for understanding how oppression manifests in data-driven research and public health contexts. To support reproducible evaluation, we release an open-sourced benchmark dataset for assessing LLMs on oppression measurement (https://github.com/chattergpt/HSO-Bench).
BibTeX
@misc{chatterjee2025assessing,
  author = {Sanjay Chatterjee and Linh Tran and Quoc Duy Nguyen and Roni Kirson and Drue Hamlin and Harvest Aquino and Hanjia Lyu and Jiebo Luo and Timothy Dye},
  title = {{Assessing Historical Structural Oppression Worldwide via Rule-Guided Prompting of Large Language Models}},
  year = {2025},
  eprint = {2509.15216},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2509.15216},
}
Figure