Classifying Peace in Global Media Using RAG and Intergroup Reciprocity

Ke Lian, Larry S. Liebovitch, Marcel Wild, Henry West, P.l. Coleman, Fang Chen, Everlyne Kimani, Kate SieckarXiv 2024 · DOI · arXiv

Abstract: This paper presents a novel approach to identifying insights of peace in global media using a Retrieval Augmented Generation (RAG) model and concepts of Positive and Negative Intergroup Reciprocity (PIR/NIR). By refining the definitions of PIR and NIR, we offer a more accurate and meaningful analysis of intergroup relations as represented in media articles. Our methodology provides insights into the dynamics that contribute to or detract from peace at a national level.

Introduction

Traditional metrics for assessing national peace, such as the Positive Peace Index (PPI) or the Human Development Index (HDI), are often constrained by lagging quantitative indicators like economic stability or governance statistics. While researchers have turned to media analysis to gain more immediate insights, standard techniques like word frequency or basic sentiment analysis often fail to capture the complexity of social dynamics. A news report might carry a "negative" sentiment due to a tragedy, yet describe a community exhibiting high levels of cooperation and support—a hallmark of social peace.

This study addresses these limitations by focusing on Intergroup Reciprocity: the specific language of cooperation and respect (Positive Intergroup Reciprocity, or PIR) versus aggression and hindrance (Negative Intergroup Reciprocity, or NIR). By analyzing how global media portrays these interactions, we can derive a more nuanced, real-time qualitative metric for peace.

Methodology: The RAG Framework

To move beyond simple keyword matching, the researchers developed a multi-stage Retrieval-Augmented Generation (RAG) pipeline. This system is designed to ground the analysis in sociological theory before processing media content.

The multi-stage RAG pipeline for peace classification.

The process follows several key steps:

  1. Embedding & Storage: Articles from a massive news corpus were processed using OpenAI’s text-embedding-3-small model. The researchers deliberately retained "raw" text—including stop words and punctuation—to allow the LLM to leverage full contextual structure.
  2. Two-Step Retrieval: Unlike standard RAG systems, this pipeline first queries a Social Science Database (containing academic works on peace and conflict) to enrich the user's query with theoretical context. This "enriched" query is then used to perform a similarity search within the Article Database (ChromaDB).
  3. LLM Synthesis & Classification: An LLM processes the retrieved academic context alongside media snippets to identify patterns of PIR or NIR. Finally, cosine similarity is used to classify 6,000 articles per country based on their alignment with these reciprocity definitions.

Datasets

The study utilized two primary data sources:

  • News On the Web (NOW) Dataset: A comprehensive collection of 700,000 articles (comprising 58 million words) across 18 countries, covering topics from hard politics to cultural reviews.
  • Social Science Database: A curated repository of academic references on peace and conflict studies, used to provide the theoretical grounding necessary for the RAG model to distinguish between complex social interactions.

Key Insights: The "Reciprocity Trap"

A critical finding during development was the failure of simple embedding searches to distinguish between PIR and NIR. Because terms like "tolerance" and "aggression" both fall under the broad semantic umbrella of "intergroup relations," the model initially struggled with valence. The researchers overcame this "Reciprocity Trap" by using the RAG pipeline to provide explicit academic context, allowing the model to differentiate between supportive and obstructive social dynamics.

This approach reveals that media serves as a social mirror; it does not merely report events but constructs a narrative of cohesion or division. High PIR alignment suggests a media landscape that reinforces mutual respect and social stability, whereas high NIR alignment highlights systemic exclusion and conflict.

Key Results and Global Rankings

The analysis produced a "Percentage Closer to PIR" ranking for the 18 countries studied, revealing significant disparities in how media portrays intergroup relations.

Normalized percentage of media articles aligning with Positive Intergroup Reciprocity (PIR) by country.
  • High PIR Alignment: New Zealand (100%), Singapore (98%), and Ireland (79%) showed the highest levels of positive reciprocity, with media narratives predominantly highlighting intergroup tolerance.
  • Moderate PIR Alignment: Countries like Sri Lanka (62%) and Bangladesh (59%) exhibited a more balanced, though often contentious, mix of supportive and obstructive dynamics.
  • Low PIR Alignment: The United States (2%) and Kenya (0%) ranked at the bottom, indicating media landscapes heavily dominated by narratives of intergroup intolerance, disrespect, and conflict.

Limitations and Future Work

While promising, the study acknowledges that these results are filtered through the lens of media outlets, which possess their own editorial biases. Consequently, the findings reflect media representation rather than a direct measure of "on-the-ground" reality.

Future research will focus on integrating non-traditional data sources, such as social media and community-level reports, to provide a more holistic view. Additionally, the team aims to develop a real-time monitoring dashboard to serve as an "early warning system" for shifts in social cohesion, potentially identifying rising tensions months before they appear in traditional socio-economic indices.

Summary updated

Abstract

This paper presents a novel approach to identifying insights of peace in global media using a Retrieval Augmented Generation (RAG) model and concepts of Positive and Negative Intergroup Reciprocity (PIR/NIR). By refining the definitions of PIR and NIR, we offer a more accurate and meaningful analysis of intergroup relations as represented in media articles. Our methodology provides insights into the dynamics that contribute to or detract from peace at a national level.
BibTeX
@misc{lian2024classifying,
  author = {Ke Lian and Larry S. Liebovitch and Marcel Wild and Henry West and P.L. Coleman and Fang Chen and Everlyne Kimani and Kate Sieck},
  title = {{Classifying Peace in Global Media Using RAG and Intergroup Reciprocity}},
  year = {2024},
  eprint = {2410.13865},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2410.13865},
}
Figure