Tiny Titans: Can Smaller Large Language Models Punch Above Their Weight in the Real World for Meeting Summarization?

Xue-Yong Fu, Md. Tahmid Rahman Laskar, Elena Khasanova, Cheng Chen, Shashi Bhushan Tn

Introduction

Deploying state-of-the-art Large Language Models (LLMs) like GPT-4 or LLaMA-2-70B in production environments presents significant challenges, particularly regarding computational cost and latency. For organizations like Dialpad that process massive volumes of business meetings, these "Larger LLMs" are often prohibitively expensive.

While traditional small models (e.g., BART or Pegasus) offer efficiency, they typically lack instruction-following capabilities—the ability to adjust a summary's length or focus based on specific user prompts (e.g., "Give me a short, concise summary"). This research investigates whether "compact" LLMs with fewer than 2 billion parameters can bridge this gap, acting as "Tiny Titans" that match the performance of giants while remaining significantly faster and cheaper to deploy.

Methodology and Datasets

The researchers evaluated two distinct groups of models:

  1. Larger LLMs (Zero-Shot): Models including GPT-3.5, PaLM-2, LLaMA-2 (7B, 13B, 70B), and Mixtral-8x7B, tested without task-specific fine-tuning.
  2. Compact LLMs (Fine-Tuned): Models under 2B parameters, such as FLAN-T5 (80M to 780M), TinyLLaMA (1.1B), and LiteLLaMA (460M), fine-tuned on meeting data.

To enable instruction-following in compact models, the researchers employed a teacher-student framework. GPT-4 was used as a "teacher" to generate high-quality reference summaries for training data across three specific prompt types: Short, Medium, and Long. The compact models were then fine-tuned to mimic these multi-length outputs.

Testing was conducted on two primary datasets:

  • In-Domain Dataset: 1,517 proprietary business meeting transcripts from Dialpad with GPT-4 generated reference summaries.
  • QMSUM-I (Instructional): A newly released benchmark created by regenerating the academic QMSUM dataset using GPT-4 to provide reference summaries for all three instruction lengths.

Key Results: The Rise of the Tiny Titan

The study's standout performer was the fine-tuned FLAN-T5-Large (780M parameters). Despite being nearly 100 times smaller than some competitors, it demonstrated remarkable efficacy.

Table 1: ROUGE score comparison between zero-shot larger LLMs and fine-tuned compact models across datasets.

As shown in the performance data above, FLAN-T5-Large achieved a ROUGE-1 score of 56.14 on the In-Domain dataset, significantly outperforming GPT-3.5 (49.55) and performing on par with the much larger fine-tuned LLaMA-2-7B. Beyond accuracy, the efficiency gains were stark: FLAN-T5-Large processed transcripts in just 4.2 seconds (vs. 15 seconds for LLaMA-2-7B) and required only 6GB of VRAM, allowing it to run on a single, low-cost NVIDIA L4 GPU.

Instruction-Following and Architecture

A critical finding was the importance of model architecture. FLAN-T5, which utilizes an encoder-decoder architecture, consistently outperformed decoder-only models like TinyLLaMA. This suggests that for summarization, architecture and the quality of instruction-tuning are more vital than raw parameter count.

Figure 1: Average ROUGE scores across different instruction types (Long, Medium, Short) for key models.

The models demonstrated a robust ability to follow length instructions. As illustrated in the figure above, the fine-tuned FLAN-T5-Large maintained high ROUGE scores across Short, Medium, and Long instructions, proving that compact models can indeed master the nuances of user-specified constraints when trained correctly.

Discussion and Limitations

While FLAN-T5-Large excelled on business meetings, the researchers identified a "Context Window Trap." The model's performance dipped on the QMSUM-I dataset because its default context window is limited to 2,048 tokens. Academic meetings in QMSUM often exceed 9,000 words, leading to information loss through truncation.

Furthermore, the study noted that in zero-shot settings, larger is not always better; LLaMA-2-70B occasionally underperformed its 7B and 13B counterparts, suggesting that massive scale can sometimes introduce noise or inefficiencies for specific niche tasks like summarization.

Conclusion

The research confirms that fine-tuned compact models—specifically FLAN-T5-Large—can serve as highly effective, cost-efficient alternatives to massive LLMs for meeting summarization. Future work will focus on overcoming context window limitations through "chapterization" (segmenting long meetings) to ensure these Tiny Titans can handle even the most extensive transcripts without losing critical information.

Summary updated

Abstract

Large Language Models (LLMs) have demonstrated impressive capabilities to solve a wide range of tasks without being explicitly fine-tuned on task-specific datasets. However, deploying LLMs in the real world is not trivial, as it requires substantial computing resources. In this paper, we investigate whether smaller, Compact LLMs are a good alternative to the comparatively Larger LLMs to address significant costs associated with utilizing LLMs in the real world. In this regard, we study the meeting summarization task in a real-world industrial environment and conduct extensive experiments by comparing the performance of fine-tuned compact LLMs (FLAN-T5, TinyLLaMA, LiteLLaMA, etc.) with zero-shot larger LLMs (LLaMA-2, GPT-3.5, PaLM-2). We observe that most smaller LLMs, even after fine-tuning, fail to outperform larger zero-shot LLMs in meeting summarization datasets. However, a notable exception is FLAN-T5 (780M parameters), which achieves performance on par with zero-shot Larger LLMs (from 7B to above 70B parameters), while being significantly smaller. This makes compact LLMs like FLAN-T5 a suitable cost-efficient LLM for real-world industrial deployment.
BibTeX
@inproceedings{fu2024tiny,
  author = {Xue-Yong Fu and Md. Tahmid Rahman Laskar and Elena Khasanova and Cheng Chen and Shashi Bhushan TN},
  title = {{Tiny Titans: Can Smaller Large Language Models Punch Above Their Weight in the Real World for Meeting Summarization?}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2024},
  doi = {10.18653/v1/2024.naacl-industry.33},
}
Figure