TL;DR
This paper tackles the high cost of using Large Language Models (LLMs) for query-based meeting summarization. By grouping multiple queries about the same meeting transcript into a single LLM prompt, the authors demonstrate a significant reduction in inference costs without sacrificing summary quality, paving the way for more practical LLM deployments.
What Problem Does This Paper Address?
Large Language Models (LLMs) are powerful tools for tasks like meeting summarization, but their real-world deployment is hampered by significant inference costs. Specifically, when a user needs summaries for multiple different aspects of a single meeting transcript (e.g., "What were the action items?" and "What was decided about the budget?"), each query typically requires a separate call to the LLM API. This leads to repeated processing of the same meeting transcript, dramatically increasing costs and making LLMs impractical for many applications. The challenge is to find a way to efficiently extract information for multiple queries from a single, potentially long, meeting transcript.
Methods
The core innovation of this work is the Multi-Query Instruction approach. Instead of sending individual queries to the LLM, the authors group all queries related to a single meeting transcript into one prompt. This prompt includes the transcript and a set of instructions asking the LLM to generate summaries for each query in a structured format (specifically, an array of JSON objects). This method aims to reduce the number of API calls by a factor equal to the number of queries per transcript. The evaluation framework involves generating responses from various LLMs using this multi-query prompt and then parsing these responses to extract the summaries for each query. Performance is then evaluated using standard metrics and human judgment.
Datasets
The primary dataset used is QMSUM (Zhong et al., 2021), a benchmark for query-focused meeting summarization. The authors created a multi-query version of QMSUM by consolidating all queries associated with a single transcript into a single prompt. This resulted in a smaller dataset with 162/35/35 instances for train/validation/test sets, compared to the original dataset's 1257/272/281 instances. Additionally, the QMSUM-I dataset (Fu et al., 2024) was used to test the generalizability of the multi-query approach across different instruction formats. The RACE-Hard dataset was also used to evaluate out-of-domain generalization.
Key Insights and Ideas
- Consolidating queries is key to cost optimization: The fundamental idea is that by sending multiple queries in one go, you avoid paying for the same context to be processed repeatedly. This is the most direct way to reduce inference costs for query-based summarization.
- Format adherence is crucial but challenging: LLMs need to output summaries in a specific, parsable format (like JSON) for automated extraction. Many LLMs struggle with consistently adhering to these complex output formats, especially when handling multiple queries.
- Closed-source LLMs generally lead, but open-source models are catching up: While top-tier closed-source models like GPT-4 and Gemini often show superior performance and format adherence, some open-source models, particularly Mistral-7B, demonstrate competitive capabilities.
- Multi-query prompts can improve summary quality: Surprisingly, in some cases, summaries generated via multi-query prompts were preferred by humans over those from single-query prompts, suggesting that the consolidated context might even aid in generating more relevant summaries.
- JSON is preferred over YAML for structured output: LLMs demonstrated better reliability in generating structured output in JSON format compared to YAML.
Key Results
- Format Following: Most closed-source LLMs achieved near 100% accuracy in generating the required JSON format, with exceptions like Claude-3-Haiku (around 80%) and GPT-4-Turbo (less than 50%). Among open-source LLMs, Mistral-7B-Instruct-v0.3 performed best at ~95%, outperforming larger closed-source models like Claude-3-Haiku and GPT-4-Turbo in this specific metric. LLaMA-2-7B-32K-Instruct showed 0% accuracy in format following.
- Summarization Performance (Zero-Shot): Gemini-1.5-Flash was the top performer across most metrics (ROUGE, BERTScore, AlignScore) in both multi-query and single-query settings. Mistral-7B-Instruct-v0.3 was the best-performing open-source model. LLMs that performed better in format following in multi-query settings sometimes showed poorer performance in single-query settings due to generating longer summaries.
- Cost Savings: For a dataset similar to QMSUM with an average of 8 queries per transcript, the multi-query approach can reduce costs by up to 8x. For example, processing 12,000 tokens for GPT-4o costs $0.06 USD; with 8 queries, this would be $0.48 USD in a single-query scenario, versus $0.06 USD with the multi-query approach.
- Human Preference: In a preference test, human evaluators preferred summaries generated via multi-query prompting over single-query prompting for the evaluated LLMs.
Limitations and Future Work
- Format Adherence Issues: A significant limitation is that many LLMs, especially open-source ones, struggle to consistently produce output in the required JSON format. The authors had to rely on a custom parser, and in some cases, summaries could not be extracted, leading to them being treated as empty responses. Future work should focus on improving LLMs' instruction-following capabilities for structured output.
- Prompt Engineering: While the authors compared various prompts, they suggest that more advanced prompt engineering techniques (like Chain-of-Thought or few-shot learning) could further enhance response generation quality.
- Dataset Limitations: The multi-query QMSUM dataset is smaller than the original, and the authors acknowledge the need for more diverse datasets for query-focused summarization.
- Future Directions: The authors plan to investigate the use of tiny LLMs for meeting summarization and explore advanced prompting techniques. They also suggest studying the impact of different output formats (e.g., YAML) and further exploring out-of-domain generalization.
Summary updated