Building Real-World Meeting Summarization Systems using Large Language Models: A Practical Perspective

Md Tahmid Rahman Laskar, Xue-Yong Fu, Cheng Chen, Shashi Bhushan Tn

TL;DR

This paper explores the practicalities of building meeting summarization systems using Large Language Models (LLMs). The authors evaluate various closed-source and open-source LLMs, finding that while closed-source models generally perform better, smaller open-source models like LLaMA-2 offer a compelling balance of performance, cost, and privacy for real-world industrial applications.

What Problem Does This Paper Address?

In today's increasingly remote work environment, meetings are more critical than ever. While recording meetings is easy, keeping up with the sheer volume of information exchanged is a significant challenge. Automated meeting summarization systems offer a solution, but traditional neural models require extensive domain-specific training data, which is often scarce in real-world industrial settings. Large Language Models (LLMs) have shown impressive zero-shot capabilities across various tasks, making them a promising avenue for meeting summarization. However, there's a lack of comparative analysis of LLMs specifically for long conversational data like meeting transcripts, especially considering practical deployment factors like cost, computational demands, and privacy. This paper aims to fill that gap by providing a practical perspective on building such systems.

Methods

The authors propose two main approaches to handle the long input sequences typical of meeting transcripts:

  • Summarization via Truncation: This straightforward method involves feeding only the first n words of the transcript to the LLM. While efficient, it risks losing crucial information from later parts of the conversation.
  • Summarization via Chapterization: This more sophisticated approach breaks down the transcript into smaller "chapters" (segments of n words). Each chapter is summarized individually, and then these chapter summaries are either concatenated, rewritten, or re-summarized to form the final summary. This method aims to retain more context than simple truncation.

They evaluate several LLMs, including GPT-3.5, GPT-4, PaLM-2, and LLaMA-2 (7B and 13B), using these methods on benchmark meeting summarization datasets.

Datasets

The paper uses the following datasets for evaluation, all of which consist of organizational meeting transcripts:

  • QMSUM (Filtered): A subset of the original QMSUM dataset, filtered to focus on overall meeting summaries rather than query-based summarization.
  • AMI: Contains scenario-based product design meetings.
  • ICSI: Consists of research-related discussions among students.

Key Insights and Ideas

  • Open-source LLMs are viable alternatives: While closed-source models like GPT-4 often achieve slightly better performance, smaller open-source models like LLaMA-2 can deliver comparable results at a significantly lower cost and with better privacy.
  • Cost and privacy are critical for industrial deployment: The high cost of API usage for closed-source models and the privacy concerns associated with sending sensitive data to third-party APIs make open-source solutions more attractive for real-world business applications.
  • Truncation can be surprisingly effective: In some scenarios, especially with shorter gold summaries, the simple truncation method performed comparably to more complex chapterization approaches, suggesting that the beginning of a transcript might hold the most salient information for summarization in certain contexts.
  • Prompt engineering matters: The choice of prompt (e.g., "long," "medium," or "short" summary) significantly impacts the length and quality of the generated summaries, highlighting the need to tailor prompts to specific user requirements.
  • LLaMA-2-7B is a strong contender: The LLaMA-2-7B model emerges as a promising candidate for industrial use due to its competitive performance, lower computational requirements, and open-source nature, making it more adaptable for fine-tuning.

Key Results

  • Performance: Closed-source models (GPT-3.5, GPT-4) generally outperformed PaLM-2 and LLaMA-2 on benchmark datasets. However, LLaMA-2 models (7B and 13B) showed competitive performance, with the 7B model performing similarly to the 13B model.
  • Cost-Effectiveness: GPT-4 was found to be significantly more costly than GPT-3.5, and PaLM-2 was also more expensive than GPT-3.5. LLaMA-2, being open-source, offers a substantial cost advantage, especially when considering deployment.
  • Inference Speed: GPT-3.5 was the fastest LLM for inference, followed by PaLM-2. GPT-4 was the slowest among the commercial models. LLaMA-2-7B took longer than GPT-3.5 but was still considered practical.
  • Chapterization vs. Truncation: The chapterization via rewriting approach generally performed best on average across datasets. However, in the QMSUM dataset, truncation was the best-performing approach. For longer summaries (AMI, ICSI), concatenation or rewriting of chapter summaries was more effective.

Limitations and Future Work

  • Dataset Limitations: The evaluation was primarily conducted on academic datasets, which may not fully represent the nuances of real-world business conversations. Future work should focus on creating domain-specific datasets.
  • Evaluation Metrics: Existing metrics like ROUGE have limitations in evaluating LLM summarization performance. Extensive human evaluation is suggested for more robust benchmarking.
  • Model Scope: The study evaluated a selection of LLMs. The authors acknowledge that many other recently proposed open-source LLMs were not included.
  • Future Directions: The authors plan to investigate the performance of LLMs by applying various optimization techniques (e.g., quantization, pruning, distillation) to further reduce memory requirements and improve inference speed. They also aim to fine-tune LLaMA-2-7B on in-domain data for production deployment.

Summary updated

Abstract

This paper studies how to effectively build meeting summarization systems for real-world usage using large language models (LLMs). For this purpose, we conduct an extensive evaluation and comparison of various closed-source and open-source LLMs, namely, GPT-4, GPT- 3.5, PaLM-2, and LLaMA-2. Our findings reveal that most closed-source LLMs are generally better in terms of performance. However, much smaller open-source models like LLaMA- 2 (7B and 13B) could still achieve performance comparable to the large closed-source models even in zero-shot scenarios. Considering the privacy concerns of closed-source models for only being accessible via API, alongside the high cost associated with using fine-tuned versions of the closed-source models, the opensource models that can achieve competitive performance are more advantageous for industrial use. Balancing performance with associated costs and privacy concerns, the LLaMA-2-7B model looks more promising for industrial usage. In sum, this paper offers practical insights on using LLMs for real-world business meeting summarization, shedding light on the trade-offs between performance and cost.
BibTeX
@inproceedings{laskar2023building,
  author = {Md Tahmid Rahman Laskar and Xue-Yong Fu and Cheng Chen and Shashi Bhushan TN},
  title = {{Building Real-World Meeting Summarization Systems using Large Language Models: A Practical Perspective}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2023},
}
Figure