Narrative-based Understanding of Society

Luc Steels, Frank van HarmelenSPIRE 2025 · DOI · OpenAlex

Abstract

As social media becomes a primary arena for public discourse, the sheer volume and fragmented nature of digital communication pose significant challenges for sociologists and policymakers. This paper presents a hybrid AI framework designed to reconstruct overarching social narratives from millions of individual social media posts. By combining Natural Language Processing (NLP) with structured Knowledge Graphs, the system maps the "fabula"—the underlying network of actors, events, and causal links—that defines public understanding of inequality. Analyzing a corpus of tweets from 2020 to 2022, we demonstrate how narratives surrounding poverty, racism, and the "digital divide" evolve in response to global crises like the COVID-19 pandemic.

Introduction

Social media platforms like Twitter (now X) serve as a real-time mirror of societal concerns. However, this mirror is often shattered into millions of disconnected fragments. While traditional sentiment analysis can gauge the "mood" of the public, it often fails to capture the structural relationships between concepts. To understand how the public perceives complex issues like inequality, we must move beyond keyword counting and toward narrative reconstruction. This research, part of the MUHAI (Meaning and Understanding in Human-centric AI) project, introduces a methodology to bridge the gap between raw, informal text and structured, narrative-based understanding.

Methodology: A Hybrid AI Framework

The proposed system operates on two distinct scales to capture both the "big picture" of social discourse and the specific nuances of individual claims.

1. Macro-Level: The Bird’s-eye Network

The first stage involves building a comprehensive co-occurrence network of entities mentioned in the discourse.

  • Entity Linking: Using spaCy for initial linguistic processing and DBpedia Spotlight, the system identifies mentions of people, organizations, and concepts, linking them to unique identifiers in the DBpedia Knowledge Graph. This ensures that variations like "the President" and "Joe Biden" are treated as a single node.
  • Network Construction and Pruning: A graph is constructed where nodes are entities and edges represent their appearance in the same tweet. To manage the complexity of massive networks, a novel local thresholding strategy is applied to prune irrelevant edges.
  • Thematic Clustering: The Louvain algorithm is employed to detect communities within the graph. These communities represent thematic "narrative clusters"—such as those focused on racial inequality, economic policy, or climate-driven poverty.

2. Micro-Level: Fine-grained Semantic Analysis

Once the broad structure is established, the system "zooms in" to understand the nature of the relationships between entities.

  • Relationship Pathways: By applying dependency parsing to the original tweets, the system extracts the linguistic paths connecting two entities (e.g., "Inflation causes Poverty").
  • Contextual Characterization: Using GloVe (Global Vectors for Word Representation) embeddings, the system identifies the adjectives and descriptors most strongly associated with an entity within a specific cluster. This reveals how the same concept (e.g., "Education") is framed differently across different narrative contexts.

Data and Case Study

The framework was tested on a massive corpus of tweets collected between January 1, 2020, and October 1, 2022, filtered for keywords related to "inequality." A representative sample of 100,000 tweets was used for the detailed network analysis. DBpedia served as the foundational Knowledge Graph, providing the "general world knowledge" necessary to ground these informal conversations in a structured taxonomy.

Key Findings: The Dynamics of Inequality

The analysis revealed several critical insights into how public narratives shift over time:

  • Narrative Absorption: The "Poverty" cluster was not static; it dynamically absorbed nodes related to "COVID-19" and "Global Warming" as the pandemic and climate crises intensified, indicating a growing public consensus on the intersectional drivers of economic hardship.
  • The Rise of the Digital Divide: In early 2020, the narrative around "educational inequality" underwent a structural shift. The system captured the emergence of the "digital divide" as a central pillar of the discourse, driven by the sudden transition to remote schooling.
  • Event-Driven Evolution: Major social shocks, such as the murder of George Floyd, led to a rapid expansion and stabilization of the "Racism" cluster. This cluster became a permanent, central fixture of the inequality narrative, rather than a fleeting spike in interest.
  • Nuanced Framing: The system successfully distinguished between general discussions of "Inflation" and its specific framing within inequality narratives, where it was consistently linked to its "disproportionate impact" on "vulnerable families."

Conclusion and Future Directions

This research demonstrates that hybrid AI—combining the statistical power of NLP with the structural logic of Knowledge Graphs—can successfully map the complex landscape of social narratives. By identifying 17 distinct thematic clusters and tracking their evolution through global crises, the framework provides a powerful tool for understanding the "pulse" of society.

Future work will focus on optimizing the pipeline for real-time analysis and developing interactive visualization tools. These tools will allow sociologists and policymakers to explore narrative networks dynamically, identifying emerging societal tensions and the underlying causal beliefs that drive public opinion.

This work was conducted as part of the MUHAI project, exploring Meaning and Understanding in Human-centric AI.

Summary updated

Abstract

This book is the third open access volume in a series reporting results of the project. The firstvolume \url{https://zenodo.org/record/6666820} surveyed how the notion of understanding is being discussed and treated in other human-centred research fields, more specifically in social brain science, social psychology, linguistics, semiotics, economics, social history and medicine. The second volume \url{https://zenodo.org/records/14199174} explored these ideas with a number of concrete case studies that are all about understanding the world. The present volume contains further case studies that now focus on understanding society. It introduces more tools for narrative understanding, and then reports on experiments to understand society through demography, through social media analysis, and through art. It also reports examples of generating social narratives from knowledge graphs.
BibTeX
@inproceedings{steels2025narrative,
  author = {Luc Steels and Frank van Harmelen},
  title = {{Narrative-based Understanding of Society}},
  booktitle = {SPIRE},
  year = {2025},
  doi = {10.5281/zenodo.15093258},
}
Figure