What Are They Talking About? A Benchmark of Knowledge-Grounded Discussion Summarization

Weixiao Zhou, Junnan Zhu, Gengyao Li, Xianfu Cheng, Xinnian Liang, Feifei Zhai, Yongbin Li

Abstract

Traditional dialogue summarization often fails when participants discuss topics they already know well, leaving "outside readers" confused by missing context and vague references. This paper introduces Knowledge-Grounded Discussion Summarization (KGDS), a new task and benchmark that requires models to generate both a background summary to bridge knowledge gaps and an opinion summary that clarifies implicit references.

Introduction

Most dialogue summarization research operates under the assumption that a transcript contains all necessary information for comprehension. However, real-world discussions—particularly in news comment sections or specialized forums—are inherently "knowledge-grounded." Participants share a common background, leading them to omit details or use ambiguous pronouns (e.g., "that match," "him") that refer to entities understood within their "insider" circle.

For an "outside reader," standard summaries of these conversations are often incomprehensible. This research identifies two primary challenges: Information Omission, where crucial context is left out as common knowledge, and Implicit Reference, where ambiguous language requires external knowledge to decode.

The KGDS Task and Paradigms

To address these challenges, the authors formalize Knowledge-Grounded Discussion Summarization (KGDS). This task requires generating two distinct outputs:

  1. Background Summary: This component retrieves or generates necessary context from a source document. It can be Extractive (EBS), involving the selection of key paragraphs, or Abstractive (ABS), which involves rewriting facts for clarity.
  2. Opinion Summary (AOS): This summarizes participant viewpoints while explicitly resolving implicit references (e.g., transforming "he scored" into "Christian Pulisic scored").
The KGDS task overview, contrasting conventional dialogue summarization with the proposed EBS-AOS and ABS-AOS paradigms.

The KGDS Benchmark

The authors introduced a specialized benchmark consisting of 100 news-discussion pairs. These pairs feature event-rich news articles coupled with human-authored discussions. The dataset includes high-quality expert annotations from NLP PhD candidates, providing multi-granularity labels:

  • Extractive labels: 432 supporting vs. 1,005 non-supporting paragraphs.
  • Abstractive labels: 1,638 key supporting atomic facts.
  • Opinion labels: 873 clear atomic opinions with 1,113 clarified implicit references.

Hierarchical Evaluation Framework

Rather than relying on traditional, often vague metrics like ROUGE, the paper proposes a hierarchical evaluation framework. Summaries are decomposed into "atomic facts" and "atomic opinions." The framework measures Coverage (inclusion of necessary information) and Focus (exclusion of irrelevant information).

The primary metric, PLQ_GM, is the geometric mean of the quality scores for both the background and opinion summaries. This ensures that a model must perform well in both context retrieval/generation and opinion synthesis to achieve a high score.

Fine-grained evaluation framework for the EBS-AOS and ABS-AOS paradigms, utilizing atomic fact and opinion verification.

Key Insights and Results

Benchmarking 12 state-of-the-art LLMs, including GPT-4o, Claude 3.5 Sonnet, and Llama-3.1-405B, revealed several critical findings:

  • Performance Gap: Even top-tier models like GPT-4o and Claude 3.5 Sonnet achieved average scores below 69%, highlighting the difficulty of the KGDS task.
  • Retrieval vs. Generation: LLMs generally perform better at retrieving relevant paragraphs (EBS) than at generating concise, abstractive background facts (ABS). Models often struggle with the "coverage-focus trade-off," either including excessive irrelevant information or missing key explanatory facts.
  • The Integration Bottleneck: There is a direct correlation between background summary quality and opinion summary accuracy. Failure to grasp the background context prevents models from correctly resolving "he/she/it" references in the discussion.
  • Implicit Reference Errors: Common failures include leaving pronouns unclarified or incorrectly clarifying them (hallucination).

Conclusion

The KGDS task represents a significant shift in summarization research, focusing on the perspective of the "outside reader." While current LLMs show promise, they struggle with the complex reasoning required to integrate external knowledge and resolve implicit references. Future work will likely focus on enhancing "knowledge-aware" integration, moving beyond simple summarization toward active information synthesis.

Summary updated

BibTeX
@misc{zhou2025what,
  author = {Weixiao Zhou and Junnan Zhu and Gengyao Li and Xianfu Cheng and Xinnian Liang and Feifei Zhai and Yongbin Li},
  title = {{What Are They Talking About? A Benchmark of Knowledge-Grounded Discussion Summarization}},
  year = {2025},
  eprint = {2505.12474},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2505.12474},
}
Figure