DiNaM: Disinformation Narrative Mining with Large Language Models

Witold Sosnowski, Arkadiusz Modzelewski, Kinga Skorupska, Adam WierzbickiEMNLP 2025

TL;DR

Disinformation campaigns are increasingly sophisticated, weaving together multiple falsehoods into cohesive narratives. Manually identifying these narratives is a monumental task. This paper introduces DiNaM, a novel algorithm that leverages Large Language Models (LLMs) to automatically mine disinformation narratives from fact-checking articles, achieving state-of-the-art results.

What Problem Does This Paper Address?

The proliferation of disinformation poses a significant threat to democratic societies and public health. While fact-checking organizations work tirelessly to debunk individual false claims, understanding the broader "narratives" – the interconnected stories that disinformation actors craft – is crucial for effective counteraction. Manually identifying these narratives is incredibly time-consuming and resource-intensive. Existing NLP research has explored narrative classification and general narrative mining, but a dedicated approach to disinformation narrative mining that systematically uncovers these complex, interconnected falsehoods has been missing. This paper aims to fill that gap by proposing an automated method to identify and extract these narratives.

Methods

DiNaM employs a multi-step methodology:

  1. Filtering Fact-Checking Articles: The process begins by filtering a dataset of fact-checking articles to select only those that exclusively rate claims as false. This binary classification task is performed using LLMs.
  2. Extracting False Information: LLMs are then used in a structured, three-step process (Extracting, Verifying, Refining) to identify and isolate specific instances of false information from the filtered articles. This approach is inspired by the Chain-of-Verification (CoVe) prompt engineering method.
  3. Clustering False Information: The extracted false information is converted into embeddings and then clustered using dimensionality reduction (UMAP) and density-based clustering (HDBSCAN) to group semantically similar pieces of information.
  4. Deriving Disinformation Narratives: Finally, an LLM is used to synthesize the clustered false information into concise, coherent disinformation narratives, adhering to specific output constraints (e.g., under 15 words, standalone, expressing the false perspective).

Datasets

  • EDMO dataset: A collection of 10,493 fact-checking articles sourced from 14 independent organizations affiliated with the European Digital Media Observatory (EDMO). This dataset was used for training and evaluating DiNaM.
  • EUDisinfoTest dataset: Used as ground truth for evaluating the performance of DiNaM's narrative derivation step. This dataset contains manually created disinformation narratives by experts.

Key Insights and Ideas

  • Narratives are more than isolated falsehoods: The paper emphasizes that disinformation operates at the narrative level, weaving together multiple false claims into a cohesive and memorable story.
  • LLMs are powerful for narrative mining: DiNaM demonstrates that LLMs, when guided by a structured, multi-step approach, can effectively identify, extract, and synthesize disinformation narratives.
  • Fact-checking articles are a rich source: Leveraging fact-checking articles provides a reliable and verified foundation for identifying false information, which is crucial for building accurate disinformation narratives.
  • Adaptability of disinformation: The analysis of mined narratives shows a strong correlation with real-world events, highlighting the dynamic and adaptable nature of disinformation campaigns.

Key Results

  • State-of-the-art performance: DiNaM achieved a Weighted Chamfer Distance (WCD) score of 0.73, outperforming general-purpose narrative mining methods (Relatio and CaNarEx) by a margin of 16.4-24.7%.
  • Effective false information extraction: DiNaM's approach to extracting false information, particularly its refinement step to remove debunking cues, led to the highest Avg WCD score (0.81) among tested prompting strategies.
  • Robust clustering: The combination of SFR-Embedding-2, UMAP, and HDBSCAN proved to be the most effective for clustering false information, achieving the highest Silhouette Score of 0.67.
  • Generalizability: DiNaM demonstrated strong generalization across diverse sources and topical categories, indicating its robustness.

Limitations and Future Work

  • LLM limitations: DiNaM inherits the semantic limitations of LLMs, which can struggle with nuanced contexts and may "hallucinate."
  • Reliance on fact-checking articles: The current approach relies on fact-checking articles, which may limit real-time detection capabilities.
  • Future work: The authors suggest exploring the integration of multimodal data (e.g., images, videos) to capture how disinformation spreads across platforms more comprehensively.

Summary updated

BibTeX
@inproceedings{sosnowski2025dinam,
  author = {Witold Sosnowski and Arkadiusz Modzelewski and Kinga Skorupska and Adam Wierzbicki},
  title = {{DiNaM: Disinformation Narrative Mining with Large Language Models}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
}
Figure