ExU: AI Models for Examining Multilingual Disinformation Narratives and Understanding their Spread

Jake Vasilakes, Zhixue Zhao, Ivan Vykopal, Michal Gregor, Martin Hyben, Carolina Scarton

TL;DR

The ExU project is developing AI models to combat multilingual disinformation by focusing on stance detection and claim retrieval. They aim to create large-scale, multilingual models that are also explainable to end-users, addressing a significant gap in current fact-checking tools.

What Problem Does This Paper Address?

Online disinformation is a pervasive issue that can have serious societal consequences. A major challenge in combating it is its spread across multiple languages, overwhelming human fact-checkers and journalists. While research has focused on fact-checking in English, there's a lack of large-scale, multilingual models for crucial tasks like stance detection (determining if a piece of content agrees or disagrees with a claim) and claim retrieval (finding relevant fact-checks for a given claim). This paper introduces the ExU project, which aims to bridge this gap by developing and evaluating novel multilingual AI methods for these tasks, with a focus on user-centric explainability.

Methods

The ExU project is developing novel methods for multilingual disinformation analysis. For stance detection, they plan to utilize multilingual transformers like Aya, which supports a wide range of languages. To address potential data scarcity in low-resource languages, they are exploring strategies such as fine-tuning with small amounts of target language data or translating low-resource languages into English to leverage existing models. A key focus is on explainability, aiming for feature attribution and rationale extraction that are consistent across languages and invariant to translation. For claim retrieval, they are employing a retrieval-augmented generation (RAG) model. This approach aims to help users efficiently identify factual claims from debunked ones by matching textual claims from online content with existing fact-checks.

Datasets

The paper mentions the MultiClaim dataset, which contains 293,169 fact-checked articles and their corresponding claims across 39 languages. This dataset is crucial for their claim retrieval task. For their initial survey, participants were recruited from the Voices Festival of Journalism and Media Literacy.

Key Insights and Ideas

  • Multilingualism is a critical bottleneck: The paper highlights that disinformation's spread across many languages is a significant hurdle for current fact-checking efforts.
  • User-centric explainability is paramount: Beyond just accurate predictions, the ExU project emphasizes the need for AI models to provide understandable explanations to end-users, which is often overlooked in purely technical AI development.
  • Bridging the mono- vs. multilingual gap in explainability: The authors acknowledge that explainability methods can perform differently between single-language and multilingual models, and plan to investigate this further.
  • RAG for efficient claim retrieval: The adoption of retrieval-augmented generation for claim retrieval suggests a promising direction for making fact-checking more efficient by directly linking content to relevant debunked claims.

Key Results

The paper reports on the initial survey conducted at the Voices Festival of Journalism and Media Literacy.

  • High multilingual exposure: 97% of the 29 survey respondents encountered content in multiple languages during their fact-checking work.
  • Language prevalence: English was the most frequently encountered language (42%), followed by French (24%), Italian (11%), Spanish (9%), and others.
  • User needs for fact-checking tools: Participants expressed a desire for content translation into their preferred language and for automatic highlighting of main arguments in posts for stance detection. For claim retrieval, they wanted summaries of fact-checks in addition to the checks themselves.

Limitations and Future Work

  • Survey bias: The authors acknowledge that the survey results might be biased towards EU languages and Italian due to the event's location in Florence. They plan to gather data from other demographics in the future.
  • Explainability gap: Previous work has shown a performance gap in explainability methods between monolingual and multilingual models, which the ExU project aims to address.
  • Low-resource language data: Obtaining sufficient data for low-resource languages remains a challenge, with strategies like fine-tuning and translation being explored.
  • Future directions: The project plans to develop and evaluate novel multilingual models for stance detection and claim retrieval, focusing on explainability and user-centric evaluation. They will also explore techniques to ensure translation invariance in explanations.

Summary updated

Abstract

Addressing online disinformation requires analysing narratives across languages to help fact-checkers and journalists sift through large amounts of data. The ExU project focuses on developing AI-based models for multilingual disinformation analysis, addressing the tasks of rumour stance classification and claim retrieval. We describe the ExU project proposal and summarise the results of a user requirements survey regarding the design of tools to support fact-checking.
BibTeX
@inproceedings{vasilakes2025exu,
  author = {Jake Vasilakes and Zhixue Zhao and Ivan Vykopal and Michal Gregor and Martin Hyben and Carolina Scarton},
  title = {{ExU: AI Models for Examining Multilingual Disinformation Narratives and Understanding their Spread}},
  booktitle = {myPublications at the University of Sheffield},
  year = {2025},
  url = {https://openreview.net/forum?id=pqua4H44yx},
}
Figure