Factuality Assessment as Modal Dependency Parsing

Jiarui Yao, Haoling Qiu, Jin Zhao, Bonan Min, Nianwen XueACL 2021

TL;DR

This paper introduces a novel approach to factuality assessment by framing it as a modal dependency parsing task. They've created the first large-scale dataset for this task and developed a joint model that simultaneously extracts events, their sources (conceivers), and the modal dependency structure, outperforming pipeline approaches.

What Problem Does This Paper Address?

In today's information-saturated world, discerning the credibility of news and online content is increasingly challenging. A crucial step in this process is understanding the factuality of events reported. This involves not only identifying the event itself but also who is reporting it (the "conceiver") and the degree of certainty they express about its occurrence. Existing methods often tackle these aspects separately, leading to potential error propagation. This paper aims to address this by developing a unified approach to extract events, their sources, and the certainty expressed, all within a structured framework.

Methods

The core of this paper's contribution lies in framing factuality assessment as modal dependency parsing. This means representing the relationships between events and their sources, along with the certainty expressed, as a structured dependency tree.

Key methodological contributions include:

  • Modal Dependency Structure (MDS): They adopt and extend the MDS framework, where nodes represent events and conceivers, and edges represent "modal strength" (level of certainty).
  • Joint Modal Dependency Parsing Model: Instead of a pipeline approach (extracting events, then conceivers, then building the structure), they propose a joint model. This model uses a pre-trained BERT encoder for contextualized token representations and then employs a multi-task learning approach to simultaneously extract events and conceivers and build the modal dependency structure. This joint approach is designed to mitigate error propagation inherent in pipeline systems.
  • Crowdsourcing Strategy: To build a large-scale dataset, they decompose the MDS annotation into four subtasks (event identification, event attachment, modal strength annotation, and conceiver superstructure construction). They detail their strategies for simplifying tasks for crowd-workers, ensuring quality through multiple annotators and payment incentives.

Datasets

The paper introduces a newly created, large-scale dataset for modal dependency parsing:

  • Modal Dependency Structure (MDS) Dataset: This dataset consists of 353 COVID-19 related news articles, annotated with 24,016 events and 2,938 conceivers. This is presented as the first large-scale corpus annotated with modal dependency structures.

The dataset was built using crowdsourcing via Amazon Mechanical Turk, with a rigorous quality control process involving multiple annotators per task and comparison against expert annotations.

Key Insights and Ideas

  • Factuality is More Than Certainty: The paper emphasizes that simply knowing the level of certainty expressed about an event is insufficient for factuality assessment. The credibility of the source (conceiver) is equally critical.
  • Joint Modeling is Key: By treating event and conceiver extraction and structure building as a single, joint task, the model can leverage interdependencies between these components, leading to improved performance, especially in the structure building phase when using automatically extracted mentions.
  • Crowdsourcing for Complex Structures is Feasible: The paper demonstrates that with careful task decomposition, clear instructions, and robust quality control, it's possible to crowdsource complex, structured annotations like modal dependency structures with high consistency.
  • Conceiver Extraction is Challenging and Linked to Structure: The results suggest that conceiver extraction is more difficult than event extraction and is closely tied to the structure building stage, as not all entities are conceivers.

Key Results

  • Dataset Size: The newly created MDS dataset is significantly larger than previous related datasets like FactBank.
  • Joint Model Performance: The joint modal dependency parsing model outperforms a pipeline model in overall end-to-end performance. Specifically, while event extraction performance was similar, the joint model showed improvements in conceiver extraction (0.2% and 2.9% on development and test sets) and parsing with automatically extracted mentions (0.6% and 1.5% on development and test sets).
  • Crowdsourcing Agreement: High agreement scores (WAWA scores ranging from 78.0% to 92.7%) were achieved among crowd-workers for the different annotation tasks, indicating the feasibility of their crowdsourcing approach. Agreement between crowd-workers and expert annotators was also substantial (e.g., Labeled Attachment Agreement of 72.1% for the overall structure).
  • Error Analysis: The system performed slightly worse than crowd-workers in identifying non-Author conceivers, particularly when the child node and parent conceiver were in different sentences, highlighting a challenging area for future improvement.

Limitations and Future Work

  • Limitations:
    • The system's performance in identifying non-Author conceivers, especially when they are in different sentences from the child node, is an area where it lags behind crowd-workers.
  • Conceiver extraction scores are generally lower than event extraction scores due to the relative scarcity of conceivers in the data.
  • Future Work:
    • The authors plan to further improve the accuracy of their parser.
  • They aim to apply the parser to conduct large-scale factuality assessments of events in news media.

Summary updated

BibTeX
@inproceedings{yao2021factuality,
  author = {Jiarui Yao and Haoling Qiu and Jin Zhao and Bonan Min and Nianwen Xue},
  title = {{Factuality Assessment as Modal Dependency Parsing}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
}
Figure