Evaluating Factual Consistency of Texts with Semantic Role Labeling

Jing Fan, Dennis Aumiller, Michael GertzACL 2023

TL;DR

This paper introduces SRLScore, a novel reference-free metric for evaluating the factual consistency of generated text, particularly for summarization tasks. By leveraging Semantic Role Labeling (SRL) to extract "fact tuples" from both input and generated text, SRLScore offers a more interpretable and adaptable approach compared to existing methods, achieving competitive results with human judgments.

What Problem Does This Paper Address?

Ensuring that automatically generated text, especially summaries, accurately reflects the information present in the original source text is a critical challenge. This "factuality" is often not a primary objective during model training, leading to "hallucinated" facts that undermine system performance and trustworthiness. While various metrics exist to automatically assess factuality, many rely on task-specific language models, making their scores difficult to interpret and potentially biased towards the models they are trained on. Existing methods also often require reference summaries, limiting their applicability. The authors aim to bridge this gap by developing a more interpretable, reference-free, and adaptable factuality evaluation metric.

Methods

The core of SRLScore lies in its two-stage process:

  1. Fact Tuple Generation: The method first uses Semantic Role Labeling (SRL) to parse sentences from both the input text and the generated summary. SRL identifies the semantic roles of words and phrases within a sentence (e.g., agent, patient, verb, time, location). This information is then used to construct "fact tuples," which are essentially structured representations of the core meaning of a sentence, independent of its syntactic formulation. For example, a sentence like "Mueller gave a book to Mary" might be represented as a tuple like (Mueller, gave, a book, Mary). The authors include seven key roles: agent, negation, relation, patient, recipient, time, and location.
  2. Fact Tuple Comparison and Scoring: Once fact tuples are generated for both the input and summary texts, SRLScore compares them to determine factual consistency. A dynamic weighting system is employed to assign importance to different roles within the tuples, allowing for adjustable scoring that can adapt to different domains. The final factuality score is computed by finding the best match for each summary fact tuple within the input fact tuples and averaging these scores. The authors also explore an optional co-reference resolution step to better link entities across sentences, though they find its performance boost is often outweighed by increased computational cost.

Datasets

The paper evaluates SRLScore on two publicly available datasets:

  • QAGS (Wang et al., 2020): This dataset consists of two splits, one derived from CNN/DailyMail and another from XSUM, containing source articles and model-generated summaries.
  • SummEval (Fabbri et al., 2021): This dataset includes synthetic summaries from various abstractive and extractive models, evaluated by multiple human annotators and experts. The authors specifically use expert ratings for factual consistency as their gold standard.

Key Insights and Ideas

  • Semantic Role Labeling for Factuality: The core innovation is using SRL to create a syntactically independent representation of facts, overcoming limitations of methods that rely on specific sentence structures.
  • Reference-Free Evaluation: By generating fact tuples from the input text itself, SRLScore eliminates the need for gold-standard summaries, making it more broadly applicable.
  • Interpretability: The "fact tuple" representation provides a degree of interpretability, allowing users to understand why a certain score was assigned, unlike black-box model-based metrics.
  • Adaptable Scoring: The dynamic weighting system allows the metric to be tuned for different domains or specific aspects of factuality.
  • Co-reference Resolution Trade-off: While co-reference resolution can improve matching, it significantly increases computational cost, suggesting a practical trade-off for real-world applications.

Key Results

  • SRLScore demonstrates competitive correlation with human judgments on factual consistency across the evaluated datasets, often performing on par with or exceeding state-of-the-art metrics.
  • The paper shows that extending fact tuples beyond simple (agent, relation, patient) triplets to include more semantic roles (like negation, recipient, time, location) generally improves correlation with human ratings.
  • Using approximate string similarity methods (like spaCy's word vectors or ROUGE) for tuple comparison generally yields better results than exact matching, especially for abstractive summaries.
  • The authors found no significant differences between several specialized factuality metrics when applying Bonferroni correction, suggesting that the field may need to re-evaluate claims of "state-of-the-art" performance on current datasets.

Limitations and Future Work

  • Runtime: The current implementation of SRLScore can be computationally expensive, especially for longer texts, though the authors note that tuple generation and comparison can be parallelized.
  • "Strike-out" Scoring: Human annotators often exhibit a "strike-out" scoring behavior (a single factual error leads to a very low score), whereas SRLScore tends to be more influenced by the majority of attributes ("bottom-up scoring"). Adopting a similar "strike-out" approach might improve correlation with human ratings.
  • Abstractive Samples: Highly abstractive summaries that retain factuality according to humans can pose challenges, potentially due to difficulties in matching synonymous expressions. Future work could explore verb sense disambiguation.
  • Dataset Limitations: The authors acknowledge that the summarization datasets used have their own limitations, often being more extractive in nature, which might not fully capture the complexities of abstractive summarization.
  • Language Transfer: While the approach is argued to be transferable to other languages, the availability of high-quality SRL models for non-English languages is currently limited.
  • Complementary Tool: SRLScore is recommended as a complementary tool to human evaluation, not a complete replacement, especially given the potential for metrics to differ from human judgments.

Summary updated

BibTeX
@inproceedings{fan2023evaluating,
  author = {Jing Fan and Dennis Aumiller and Michael Gertz},
  title = {{Evaluating Factual Consistency of Texts with Semantic Role Labeling}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
}
Figure