Tell me what I need to know: Exploring LLM-based (Personalized) Abstractive Multi-Source Meeting Summarization

Frederic Kirstein, Terry Ruas, Robert Kratel, Bela GippEMNLP 2024

TL;DR

This paper introduces a novel three-stage pipeline for abstractive meeting summarization that leverages multiple Large Language Models (LLMs) and Retrieval Augmented Generation (RAG) to incorporate supplementary materials. This multi-source approach significantly enhances summary quality and informativeness. Furthermore, the authors introduce a personalization protocol that tailors summaries to individual participants' needs and preferences, improving relevance and informativeness.

What Problem Does This Paper Address?

Existing meeting summarization tools, often relying solely on transcripts, struggle to generate personalized and contextually rich summaries. They often fail to identify crucial information, understand the full meeting content, and tailor outputs to individual user needs. While incorporating supplementary materials like slides is a promising direction, current methods are limited by LLMs' context window sizes and the complexity of handling multi-source information, leading to issues with redundancy, contradiction, and coherence. There's a clear need for improved comprehension and personalization in meeting summarization.

Methods

The paper proposes a three-stage pipeline that mimics human summarization:

  1. Gap Identification: An LLM analyzes the meeting transcript to identify passages lacking context and generates questions about the missing information. This stage is informed by research on knowledge gap detection and LLM reasoning capabilities.
  2. Information Inferring: A Retrieval Augmented Generation (RAG) framework uses these generated questions to retrieve relevant details from supplementary documents. These retrieved answers are then inserted into the transcript as comments. This leverages RAG's ability to efficiently filter and retrieve information from large document collections.
  3. Summarization: A final LLM generates an abstractive summary from the enriched transcript, now containing inferred contextual details.

For personalization, an additional LLM extracts participant characteristics (personality traits, project interests, knowledge level) from the transcript. This persona information is then used to guide the gap identification and summarization stages, tailoring the output to the individual's perspective and needs.

Datasets

The primary dataset used is MS-AMI, an adapted version of the AMI dataset. It comprises 125 staged business meetings with processed supplementary content, including whiteboard drawings, slides, and notes. The authors generated this dataset by using GPT-4 for OCR and image description, and Aspose for document text extraction.

Key Insights and Ideas

  • Structured Inclusion of Inferred Details is Key: Simply appending supplementary documents to transcripts is less effective than strategically inferring and integrating relevant information into the transcript itself. This structured approach significantly enhances context understanding and summary quality.
  • Multi-Source Summarization Boosts Quality: Incorporating supplementary materials, even with a limited number of top-ranked sources, demonstrably improves summary informativeness and relevance compared to single-source summarization.
  • Personalization Matters: Tailoring summaries to individual participants' roles, interests, and knowledge levels leads to more informative and relevant outputs, improving user engagement and utility.
  • Smaller Models Can Be Viable: While larger models like GPT-4 offer superior performance, smaller, more efficient models (e.g., Phi-3 mini) can provide good-quality personalized summaries, especially for on-device or cost-constrained scenarios, though they may require additional robustness measures.
  • Persona Extraction is Effective: Automatically extracting participant personas from transcripts allows for a more targeted and personalized summarization experience.

Key Results

  • The multi-source approach improved summary informativeness by approximately 9% and produced more content-rich outputs.
  • Personalization improved informativeness by approximately 10%.
  • LLM-based metrics achieved high accuracy compared to human labels for relevance (87.3%), informativeness (92.4%), factuality (85.7%), and overall quality (93.6%).
  • For personalized summaries, accuracy scores were also high: relevance (91.5%), informativeness (89.8%), and overall quality (87.8%).
  • The G-infer pipeline (multi-source with RAG) showed significant improvements in informativeness, factuality, and overall quality compared to baselines.
  • Personalization (P-per) improved informativeness and relevance scores compared to a simple personalized baseline.
  • When comparing different LLMs, GPT-4 generally offered the best performance, while Phi-3 mini provided a cost-effective alternative with similar quality.

Limitations and Future Work

  • Dataset Size: While comparable to the original AMI dataset, the MS-AMI dataset of 125 samples might still be considered small for some applications.
  • Model Optimization: Optimizing prompts across different LLM families for each stage of the pipeline presents challenges and can lead to performance variations.
  • Future Work: The authors suggest exploring more sophisticated approaches, such as multi-agent discussions for retrieval and personalization, and developing dynamic functions to identify the optimal amount of resources to consider. They also propose the development of critique models to further refine generated summaries.

Summary updated

BibTeX
@inproceedings{kirstein2024tell,
  author = {Frederic Kirstein and Terry Ruas and Robert Kratel and Bela Gipp},
  title = {{Tell me what I need to know: Exploring LLM-based (Personalized) Abstractive Multi-Source Meeting Summarization}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2024},
}
Figure