Re-FRAME the Meeting Summarization SCOPE: Fact-Based Summarization and Personalization via Questions

Frederic Kirstein, Sonu Kumar, Terry Ruas, Bela GippEMNLP 2025

TL;DR

This paper introduces FRAME, a novel fact-based pipeline for meeting summarization that reframes the task as semantic enrichment. It also proposes SCOPE, a reason-out-loud protocol for personalized summaries, and P-MESA, a reference-free metric to evaluate personalization quality. These contributions aim to improve the faithfulness, control, and personalization of meeting summaries.

What Problem Does This Paper Address?

Summarizing meetings using large language models (LLMs) is challenging due to the inherent nature of conversations: information is scattered across turns, utterances depend on long-range context, and salience varies per listener. Current LLM-based systems often produce summaries with hallucinations, omissions, and irrelevancies. This is attributed to approaches that treat conversations as linear text, compressing form without reconstructing meaning. The paper argues that a fundamental mismatch exists because current methods condense information without truly reconstructing the underlying semantic structure, leading to a lack of control, faithfulness, and personalization.

Methods

The paper proposes FRAME, a modular pipeline that treats summarization as a semantic enrichment task. It operates in four stages:

  1. Fact Identification: Extracts self-contained, verifiable facts (statement-context tuples) from the transcript to filter out filler content. This stage includes fact extraction and verification to ensure accuracy and completeness.
  2. Note-Taking: Scores the relevance of extracted facts and groups them thematically. This helps in reducing information overload and controlling content.
  3. Organization: Creates a structured outline to guide the summary, organizing facts by thematic relationships and prioritizing key decisions and discussions.
  4. Summarization: Enriches the outline into an abstractive summary by using the extracted facts and their contexts, ensuring the summary is grounded and avoids unsupported content.

For personalization, the paper introduces SCOPE, a reason-out-loud protocol. SCOPE guides an LLM to answer a questionnaire about the reader's goals, expertise, and understanding before fact selection. This creates an explicit reasoning trace to ground content selection and improve personalization.

To evaluate personalized summaries, the paper proposes P-MESA, a reference-free, multi-dimensional metric that quantifies personalization quality across seven dimensions: factuality, completeness, relevance, goal alignment, priority structuring, knowledge-fit, and contextual framing.

Datasets

The paper evaluates its methods on two datasets:

  • QMSum: An established benchmark containing academic, council, and parliamentary meetings.
  • FAME: A synthetic dataset comprising 500 English and 300 German meetings, generated by LLM agents, covering 14 meeting formats and 28 topics, and designed to simulate realistic meeting dynamics with conversational interruptions.

Key Insights and Ideas

  • Fact-based summarization as enrichment: Reframing summarization as an enrichment task, rather than simple compression, leads to improved content understanding, reduced hallucinations, and better coherence.
  • The importance of structured reasoning for personalization: SCOPE's explicit reason-out-loud protocol significantly outperforms implicit methods like role-playing or reader-tailoring by grounding personalization in a detailed understanding of the reader's intent.
  • A robust metric for personalization is crucial: P-MESA provides a much-needed reference-free evaluation framework that aligns well with human judgment, enabling better assessment of personalized summaries.
  • Global context is key for fact representation: Using statement-context tuples that incorporate broader meeting context is essential for accurate fact interpretation, resolving references, and understanding implications.
  • Modular pipeline design is beneficial: Each stage of the FRAME pipeline (fact identification, note-taking, organization, summarization) plays a critical role in maintaining summary quality, and collapsing stages leads to degradation.

Key Results

  • FRAME: Consistently outperforms baselines on MESA metrics, significantly reducing hallucination (e.g., from 3 to 1 on QMSum) and irrelevance. It achieves a higher compression ratio (~80%) while maintaining quality.
  • SCOPE: Improves personalization across all seven P-MESA dimensions, leading to better relevance, knowledge-level fit, and goal alignment compared to role-playing and reader-tailoring baselines. It also reduces hallucination and omission in personalized summaries.
  • P-MESA: Achieves high detection accuracy (B-ACC > 89%) and strong rank correlation with human severity ratings (Spearman's p up to 0.81), demonstrating its reliability as a proxy for human judgment.
  • Cross-model and cross-domain generalization: FRAME shows robust performance across different LLM backbones (GPT-4o, Gemini, Llama, Gemma) and extends its benefits to diverse document types beyond meetings, improving repetition and omission.

Limitations and Future Work

  • LLM dependency: The performance of FRAME and SCOPE is significantly dependent on the capabilities of the underlying LLM. Smaller models or those with limited context windows may yield less accurate results.
  • Computational requirements: The multi-stage pipeline, especially fact extraction and verification, incurs higher inference time and computational costs compared to end-to-end approaches.
  • Dataset coverage: While QMSum and FAME offer diverse meeting types, they may not capture all specialized domains (e.g., medical consultations, legal).
  • Future work: The authors suggest extending FRAME and SCOPE to multilingual applications, multi-source integration, and other domains. They also highlight the potential for further optimization of fact retention thresholds and the development of more robust quality control for less capable models.

Summary updated

BibTeX
@inproceedings{kirstein2025re,
  author = {Frederic Kirstein and Sonu Kumar and Terry Ruas and Bela Gipp},
  title = {{Re-FRAME the Meeting Summarization SCOPE: Fact-Based Summarization and Personalization via Questions}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
}
Figure