Building Real-World Meeting Summarization Systems using Large Language Models: A Practical Perspective

Md Tahmid Rahman Laskar, Xue-Yong Fu, Cheng Chen, Shashi Bhushan Tn

Abstract

This paper studies how to effectively build meeting summarization systems for real-world usage using large language models (LLMs). For this purpose, we conduct an extensive evaluation and comparison of various closed-source and open-source LLMs, namely, GPT-4, GPT-3.5, PaLM-2, and LLaMA-2. Our findings reveal that most closed-source LLMs are generally better in terms of performance. However, much smaller open-source models like LLaMA-2 (7B and 13B) could still achieve performance comparable to the large closed-source models even in zero-shot scenarios. Considering the privacy concerns of closed-source models for only being accessible via API, alongside the high cost associated with using fine-tuned versions of the closed-source models, the open-source models that can achieve competitive performance are more advantageous for industrial use. Balancing performance with associated costs and privacy concerns, the LLaMA-2-7B model looks more promising for industrial usage. In sum, this paper offers practical insights on using LLMs for real-world business meeting summarization, shedding light on the trade-offs between performance and cost.

Introduction

Summarizing meetings is a critical yet challenging NLP task. Transcripts are typically long, unstructured, and often contain errors from Speech-to-Text (STT) systems. Furthermore, corporate meetings frequently involve sensitive data, making privacy a paramount concern. While Large Language Models (LLMs) offer powerful zero-shot capabilities that bypass the need for massive annotated datasets, their deployment in industrial settings faces three primary hurdles: limited context windows, data privacy risks associated with third-party APIs, and the high economic costs of high-performance models like GPT-4.

Methodology

The study evaluates four major LLM families: GPT-4, GPT-3.5, PaLM-2, and LLaMA-2 (7B and 13B). To address the context window limitations of these models when processing long meeting transcripts, two primary strategies were implemented:

  1. Summarization via Truncation: Feeding the model only the initial segment of the transcript (e.g., the first 2500 words).
  2. Summarization via Chapterization: Segmenting the transcript into "chapters," summarizing each, and then merging them using three techniques: direct concatenation, re-summarizing the summaries, or re-writing them into a narrative.
Summarization strategies: (i) Truncation vs (ii) Chapterization with merging techniques.

The models were benchmarked across several datasets, including AMI (product design), ICSI (academic discussions), and filtered versions of QMSUM and MeetingBank, focusing on overall meeting summaries.

Results and Performance Comparison

The evaluation utilized standard metrics such as ROUGE and BERTScore to assess summary quality across the different models.

Performance comparison of LLMs across ROUGE and BERTScore metrics.

Model Performance

GPT-4 consistently emerged as the "gold standard" for quality across most metrics. However, a significant finding was the underperformance of PaLM-2, which often trailed behind the much smaller LLaMA-2-7B model. Interestingly, LLaMA-2-7B and LLaMA-2-13B showed nearly identical performance, suggesting that the 7B variant is more efficient for this specific task.

Strategy Effectiveness

A surprising "lead bias" was observed: simple truncation often performed comparably to more complex chapterization methods. This is likely due to human-annotated academic datasets being biased toward the beginning of meetings. Among chapterization merging techniques, "re-writing" showed promise, with GPT-3.5 occasionally outperforming GPT-4 on specific datasets when using this approach.

Operational Trade-offs

Inference speed and cost are critical for real-world deployment. GPT-3.5 was the fastest (2.5s per transcript), while GPT-4 was significantly slower (up to 40s under load) and roughly 25 times more expensive. LLaMA-2-7B, running on a single NVIDIA L4 GPU, took approximately 15s, offering a balance of speed and local hosting capabilities.

Discussion and Industrial Implications

For industrial applications, the LLaMA-2-7B model represents a "sweet spot." It provides competitive performance while allowing for local hosting, which eliminates the privacy risks of sending sensitive transcripts to external APIs. Furthermore, its ability to run on modest hardware makes it economically viable for high-volume processing.

While current metrics like ROUGE provide a baseline, they may not fully capture the "usefulness" of a summary in a business context. Future work should focus on human evaluation and the development of datasets reflecting real-world corporate conversations. Additionally, optimization techniques such as quantization and pruning could further enhance the efficiency of open-source models for production environments.

Conclusion

Transitioning meeting summarization from research to production requires balancing quality, cost, and privacy. While closed-source models like GPT-4 lead in performance, open-source alternatives like LLaMA-2-7B offer a more practical and secure path for enterprise deployment, providing sufficient quality at a fraction of the cost and with full data sovereignty.

Summary updated

Abstract

This paper studies how to effectively build meeting summarization systems for real-world usage using large language models (LLMs). For this purpose, we conduct an extensive evaluation and comparison of various closed-source and open-source LLMs, namely, GPT-4, GPT- 3.5, PaLM-2, and LLaMA-2. Our findings reveal that most closed-source LLMs are generally better in terms of performance. However, much smaller open-source models like LLaMA- 2 (7B and 13B) could still achieve performance comparable to the large closed-source models even in zero-shot scenarios. Considering the privacy concerns of closed-source models for only being accessible via API, alongside the high cost associated with using fine-tuned versions of the closed-source models, the opensource models that can achieve competitive performance are more advantageous for industrial use. Balancing performance with associated costs and privacy concerns, the LLaMA-2-7B model looks more promising for industrial usage. In sum, this paper offers practical insights on using LLMs for real-world business meeting summarization, shedding light on the trade-offs between performance and cost.
BibTeX
@inproceedings{laskar2023building,
  author = {Md Tahmid Rahman Laskar and Xue-Yong Fu and Cheng Chen and Shashi Bhushan TN},
  title = {{Building Real-World Meeting Summarization Systems using Large Language Models: A Practical Perspective}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2023},
}
Figure