Abstract
Large Language Models (LLMs) have demonstrated impressive capabilities to solve a wide range of tasks without being explicitly fine-tuned on task-specific datasets. However, deploying LLMs in the real world is not trivial, as it requires substantial computing resources. In this paper, we investigate whether smaller, compact LLMs are a good alternative to the comparatively Larger LLMs to address significant costs associated with utilizing LLMs in the real world. In this regard, we study the meeting summarization task in a real-world industrial environment and conduct extensive experiments by comparing the performance of fine-tuned compact LLMs (e.g., FLAN-T5, TinyLLaMA, LiteLLaMA) with zero-shot larger LLMs (e.g., LLaMA-2, GPT-3.5, PaLM-2). We observe that most smaller LLMs, even after fine-tuning, fail to outperform larger zero-shot LLMs in meeting summarization datasets. However, a notable exception is FLAN-T5 (780M parameters), which performs on par or even better than many zero-shot Larger LLMs (from 7B to above 70B parameters), while being significantly smaller. This makes compact LLMs like FLAN-T5 a suitable cost-efficient solution for real-world industrial deployment.
Introduction
In the current landscape of Large Language Models (LLMs), the prevailing wisdom often suggests that "bigger is better." However, for industrial applications, massive models present a "triple threat" of challenges: prohibitive costs at scale, high latency that degrades user experience, and a lack of flexibility in following specific formatting or length instructions. This study investigates whether compact LLMs (under 2 billion parameters) can be fine-tuned to match or exceed the performance of industry giants like GPT-3.5 and LLaMA-2-70B specifically for the task of meeting summarization.
Methodology
The research evaluates two distinct groups of models across two primary datasets:
- Larger Zero-Shot LLMs: Models including GPT-3.5, PaLM-2, LLaMA-2 (7B, 13B, 70B), and Mixtral-8x7B, tested without task-specific training.
- Compact Fine-Tuned LLMs: Models including FLAN-T5 (Small, Base, Large), TinyLLaMA (1.1B), and LiteLLaMA (460M), fine-tuned on meeting-specific data.
Datasets
The models were tested on an In-Domain Dataset consisting of 1,517 real-world business meeting transcripts and QMSUM-I, an instructional version of the public QMSUM dataset. To ensure high-quality benchmarks, GPT-4 was utilized to generate "gold standard" reference summaries for three distinct lengths: Short, Medium, and Long.
Performance Analysis
The experimental results reveal a significant outlier in the compact model category. While most small models like TinyLLaMA struggled to compete with larger zero-shot counterparts, FLAN-T5-Large (780M parameters) emerged as a "Tiny Titan."
On the In-Domain dataset, the fine-tuned FLAN-T5-Large achieved a ROUGE-1 score of 56.14, notably outperforming zero-shot GPT-3.5 (49.55) and Mixtral-8x7B (51.99). This performance is particularly impressive given that FLAN-T5-Large is roughly 10 to 100 times smaller than the models it surpassed.
Instruction-Aware Summarization
A key requirement for real-world deployment is the ability to follow length-based instructions. The researchers tested how models adapted when asked for "Short," "Medium," or "Long" summaries.
As shown in the figure above, FLAN-T5-Large-FT demonstrated strong performance across all instruction types in the In-Domain setting, with a particular "sweet spot" for medium-length summaries. While larger models like GPT-3.5 and LLaMA-2 showed greater stability across the extremes of short and long instructions, FLAN-T5's overall ROUGE scores remained superior in the target domain.
Discussion and Efficiency
Beyond raw accuracy, the compact FLAN-T5-Large offers substantial operational advantages:
- Latency: It processed transcripts in an average of 4.2 seconds, compared to 15 seconds for LLaMA-2-7B.
- Hardware Requirements: Requiring only 6GB of VRAM, it can run on entry-level consumer GPUs, whereas LLaMA-2-7B and 13B require 24GB and 48GB respectively.
However, a primary limitation is the context window. FLAN-T5's 2,048-token limit resulted in lower performance on the QMSUM-I dataset, where transcripts average ~9,000 words. In these instances, larger models with 4,000+ token windows maintained an advantage by processing more of the source text.
Conclusion
Fine-tuned compact models, specifically FLAN-T5-Large, represent a highly viable, cost-efficient alternative to massive zero-shot LLMs for domain-specific tasks like meeting summarization. While context window limitations remain a hurdle for long-form transcripts, the efficiency and instruction-following capabilities of these "Tiny Titans" make them ideal for industrial deployment.
Summary updated