The Intended Uses of Automated Fact-Checking Artefacts: Why, How and Who

Michael Sejr Schlichtkrull, Nedjma Ousidhoum, Andreas VlachosEMNLP 2023 · DOI

Introduction: The "Hammer" Problem in AI

In the rapidly evolving field of Natural Language Processing (NLP), "automated fact-checking" (AFC) has emerged as a critical frontier. Researchers have developed a vast array of "artefacts"—including models, datasets, and applications—aimed at combating the global misinformation crisis. However, a fundamental question often remains unanswered: how are these tools actually intended to be used?

Drawing on Heidegger’s philosophy, the authors of this study argue that a technological tool is only truly understood through its intended use. Just as a hammer is defined by the act of hammering, an AFC tool must be defined by its practical application. Currently, many AFC papers suffer from a "vagueness gap," where noble goals (e.g., "fighting fake news") are stated without specifying who will use the tool or how it will integrate into real-world workflows. This lack of clarity leads to overclaiming and prevents meaningful evaluation of whether the technology actually achieves its societal purpose.

Methodology: Mapping the Narrative Landscape

To investigate this issue, the researchers conducted a content analysis of 100 highly-cited papers in the AFC field. They developed an annotation scheme to extract "epistemic elements"—the core building blocks of a research narrative:

  • Stakeholders: Including Data Subjects (those whose information is checked), Data Actors (the end-users), and Model Owners (those who deploy the system).
  • Means: Distinguishing between Modeling Means (the technical ML task, such as classification) and Application Means (the real-world action, such as providing veracity scores).
  • Ends: The ultimate societal goal (e.g., limiting the spread of misinformation).
The conceptual framework of epistemic elements used to analyze AFC research narratives.

A narrative is classified as vague if it fails to establish a clear, logical connection between the technical means and the intended real-world application.

Key Insights: The Prevalence of Vagueness

The analysis reveals that vagueness is the default state in AFC research. Over half (56%) of the analyzed papers rely on "vague narratives."

  • Vague Identification: The most common narrative (31%) involves claiming a tool will identify misinformation without explaining the subsequent action.
  • The Missing "Who": In these vague papers, "Data Actors"—the actual humans intended to use the tools—are almost entirely absent. This creates a "God Trick" perspective, where research is presented as a detached, universal solution without accountability for specific real-world contexts.
  • Mismatched Means and Ends: There is a significant disconnect between technical tasks and stated goals. For instance, while many papers aim to "supplant" human fact-checkers, 81% rely on simple binary classification. This ignores the complex editorial reasoning and provenance work that defines professional fact-checking.

Quantifying the Narrative Landscape

The study provides a quantitative breakdown of how AFC research is framed. While 78.1% of papers aim to limit misinformation, the majority (79.3%) focus narrowly on veracity scoring as their primary technical contribution.

Frequency of different intended use narratives across the 100 analyzed AFC papers.

Furthermore, even among papers with "clear" narratives, only 18% provided empirical evidence—such as user studies—to demonstrate that their proposed application is feasible or effective in practice. There is also a notable stakeholder bias: when actors are mentioned, 62% are professional journalists, largely ignoring the needs of ordinary citizens or social media moderators.

Conclusion and Future Directions

While there is a positive trend toward tools that support human-in-the-loop processes rather than providing raw veracity scores, the field still faces a significant "feasibility gap." To move forward, the authors suggest broadening stakeholders beyond journalists to include ordinary users and community-driven initiatives. They also call for empirical use-testing to ensure that technical improvements in accuracy actually translate to a reduction in misinformation in practice.

Summary updated

BibTeX
@inproceedings{schlichtkrull2023intended,
  author = {Michael Sejr Schlichtkrull and Nedjma Ousidhoum and Andreas Vlachos},
  title = {{The Intended Uses of Automated Fact-Checking Artefacts: Why, How and Who}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2023},
  doi = {10.18653/v1/2023.findings-emnlp.577},
}
Figure