Tiny Titans: Can Smaller Large Language Models Punch Above Their Weight in the Real World for Meeting Summarization?

Xue-Yong Fu, Md Tahmid Rahman Laskar, Elena Khasanova, Cheng Chen, Shashi Bhushan Tn

Abstract

Large Language Models (LLMs) have demonstrated impressive capabilities to solve a wide range of tasks without being explicitly fine-tuned on task-specific datasets. However, deploying LLMs in the real world is not trivial, as it requires substantial computing resources. In this paper, we investigate whether smaller, compact LLMs are a good alternative to the comparatively Larger LLMs to address significant costs associated with utilizing LLMs in the real world. In this regard, we study the meeting summarization task in a real-world industrial environment and conduct extensive experiments by comparing the performance of fine-tuned compact LLMs (e.g., FLAN-T5, TinyLLaMA, LiteLLaMA) with zero-shot larger LLMs (e.g., LLaMA-2, GPT-3.5, PaLM-2). We observe that most smaller LLMs, even after fine-tuning, fail to outperform larger zero-shot LLMs in meeting summarization datasets. However, a notable exception is FLAN-T5 (780M parameters), which performs on par or even better than many zero-shot Larger LLMs (from 7B to above 70B parameters), while being significantly smaller. This makes compact LLMs like FLAN-T5 a suitable cost-efficient solution for real-world industrial deployment.

Introduction

In the current landscape of Large Language Models (LLMs), the prevailing wisdom often suggests that "bigger is better." However, for industrial applications, massive models present a "triple threat" of challenges: prohibitive costs at scale, high latency that degrades user experience, and a lack of flexibility in following specific formatting or length instructions. This study investigates whether compact LLMs (under 2 billion parameters) can be fine-tuned to match or exceed the performance of industry giants like GPT-3.5 and LLaMA-2-70B specifically for the task of meeting summarization.

Methodology

The research evaluates two distinct groups of models across two primary datasets:

  1. Larger Zero-Shot LLMs: Models including GPT-3.5, PaLM-2, LLaMA-2 (7B, 13B, 70B), and Mixtral-8x7B, tested without task-specific training.
  2. Compact Fine-Tuned LLMs: Models including FLAN-T5 (Small, Base, Large), TinyLLaMA (1.1B), and LiteLLaMA (460M), fine-tuned on meeting-specific data.

Datasets

The models were tested on an In-Domain Dataset consisting of 1,517 real-world business meeting transcripts and QMSUM-I, an instructional version of the public QMSUM dataset. To ensure high-quality benchmarks, GPT-4 was utilized to generate "gold standard" reference summaries for three distinct lengths: Short, Medium, and Long.

Performance Analysis

The experimental results reveal a significant outlier in the compact model category. While most small models like TinyLLaMA struggled to compete with larger zero-shot counterparts, FLAN-T5-Large (780M parameters) emerged as a "Tiny Titan."

Comparison of ROUGE scores across zero-shot and fine-tuned models on In-Domain and QMSUM-I datasets.

On the In-Domain dataset, the fine-tuned FLAN-T5-Large achieved a ROUGE-1 score of 56.14, notably outperforming zero-shot GPT-3.5 (49.55) and Mixtral-8x7B (51.99). This performance is particularly impressive given that FLAN-T5-Large is roughly 10 to 100 times smaller than the models it surpassed.

Instruction-Aware Summarization

A key requirement for real-world deployment is the ability to follow length-based instructions. The researchers tested how models adapted when asked for "Short," "Medium," or "Long" summaries.

Average ROUGE scores for different instruction types (Short, Medium, Long) across key models.

As shown in the figure above, FLAN-T5-Large-FT demonstrated strong performance across all instruction types in the In-Domain setting, with a particular "sweet spot" for medium-length summaries. While larger models like GPT-3.5 and LLaMA-2 showed greater stability across the extremes of short and long instructions, FLAN-T5's overall ROUGE scores remained superior in the target domain.

Discussion and Efficiency

Beyond raw accuracy, the compact FLAN-T5-Large offers substantial operational advantages:

  • Latency: It processed transcripts in an average of 4.2 seconds, compared to 15 seconds for LLaMA-2-7B.
  • Hardware Requirements: Requiring only 6GB of VRAM, it can run on entry-level consumer GPUs, whereas LLaMA-2-7B and 13B require 24GB and 48GB respectively.

However, a primary limitation is the context window. FLAN-T5's 2,048-token limit resulted in lower performance on the QMSUM-I dataset, where transcripts average ~9,000 words. In these instances, larger models with 4,000+ token windows maintained an advantage by processing more of the source text.

Conclusion

Fine-tuned compact models, specifically FLAN-T5-Large, represent a highly viable, cost-efficient alternative to massive zero-shot LLMs for domain-specific tasks like meeting summarization. While context window limitations remain a hurdle for long-form transcripts, the efficiency and instruction-following capabilities of these "Tiny Titans" make them ideal for industrial deployment.

Summary updated

Abstract

Large Language Models (LLMs) have demonstrated impressive capabilities to solve a wide range of tasks without being explicitly fine-tuned on task-specific datasets. However, deploying LLMs in the real world is not trivial, as it requires substantial computing resources. In this paper, we investigate whether smaller, compact LLMs are a good alternative to the comparatively Larger LLMs2 to address significant costs associated with utilizing LLMs in the real world. In this regard, we study the meeting summarization task in a real-world industrial environment and conduct extensive experiments by comparing the performance of fine-tuned compact LLMs (e.g., FLAN-T5, TinyLLaMA, LiteLLaMA) with zero-shot larger LLMs (e.g., LLaMA-2, GPT-3.5, PaLM-2). We observe that most smaller LLMs, even after fine-tuning, fail to outperform larger zero-shot LLMs in meeting summarization datasets. However, a notable exception is FLAN-T5 (780M parameters), which performs on par or even better than many zero-shot Larger LLMs (from 7B to above 70B parameters), while being significantly smaller. This makes compact LLMs like FLAN-T5 a suitable cost-efficient solution for real-world industrial deployment.
BibTeX
@inproceedings{fu2024tiny,
  author = {Xue-Yong Fu and Md Tahmid Rahman Laskar and Elena Khasanova and Cheng Chen and Shashi Bhushan TN},
  title = {{Tiny Titans: Can Smaller Large Language Models Punch Above Their Weight in the Real World for Meeting Summarization?}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2024},
  doi = {10.18653/v1/2024.naacl-industry.33},
}
Figure