Introduction
Deploying state-of-the-art Large Language Models (LLMs) like GPT-4 or LLaMA-2-70B in production environments presents significant challenges, particularly regarding computational cost and latency. For organizations like Dialpad that process massive volumes of business meetings, these "Larger LLMs" are often prohibitively expensive.
While traditional small models (e.g., BART or Pegasus) offer efficiency, they typically lack instruction-following capabilities—the ability to adjust a summary's length or focus based on specific user prompts (e.g., "Give me a short, concise summary"). This research investigates whether "compact" LLMs with fewer than 2 billion parameters can bridge this gap, acting as "Tiny Titans" that match the performance of giants while remaining significantly faster and cheaper to deploy.
Methodology and Datasets
The researchers evaluated two distinct groups of models:
- Larger LLMs (Zero-Shot): Models including GPT-3.5, PaLM-2, LLaMA-2 (7B, 13B, 70B), and Mixtral-8x7B, tested without task-specific fine-tuning.
- Compact LLMs (Fine-Tuned): Models under 2B parameters, such as FLAN-T5 (80M to 780M), TinyLLaMA (1.1B), and LiteLLaMA (460M), fine-tuned on meeting data.
To enable instruction-following in compact models, the researchers employed a teacher-student framework. GPT-4 was used as a "teacher" to generate high-quality reference summaries for training data across three specific prompt types: Short, Medium, and Long. The compact models were then fine-tuned to mimic these multi-length outputs.
Testing was conducted on two primary datasets:
- In-Domain Dataset: 1,517 proprietary business meeting transcripts from Dialpad with GPT-4 generated reference summaries.
- QMSUM-I (Instructional): A newly released benchmark created by regenerating the academic QMSUM dataset using GPT-4 to provide reference summaries for all three instruction lengths.
Key Results: The Rise of the Tiny Titan
The study's standout performer was the fine-tuned FLAN-T5-Large (780M parameters). Despite being nearly 100 times smaller than some competitors, it demonstrated remarkable efficacy.
As shown in the performance data above, FLAN-T5-Large achieved a ROUGE-1 score of 56.14 on the In-Domain dataset, significantly outperforming GPT-3.5 (49.55) and performing on par with the much larger fine-tuned LLaMA-2-7B. Beyond accuracy, the efficiency gains were stark: FLAN-T5-Large processed transcripts in just 4.2 seconds (vs. 15 seconds for LLaMA-2-7B) and required only 6GB of VRAM, allowing it to run on a single, low-cost NVIDIA L4 GPU.
Instruction-Following and Architecture
A critical finding was the importance of model architecture. FLAN-T5, which utilizes an encoder-decoder architecture, consistently outperformed decoder-only models like TinyLLaMA. This suggests that for summarization, architecture and the quality of instruction-tuning are more vital than raw parameter count.
The models demonstrated a robust ability to follow length instructions. As illustrated in the figure above, the fine-tuned FLAN-T5-Large maintained high ROUGE scores across Short, Medium, and Long instructions, proving that compact models can indeed master the nuances of user-specified constraints when trained correctly.
Discussion and Limitations
While FLAN-T5-Large excelled on business meetings, the researchers identified a "Context Window Trap." The model's performance dipped on the QMSUM-I dataset because its default context window is limited to 2,048 tokens. Academic meetings in QMSUM often exceed 9,000 words, leading to information loss through truncation.
Furthermore, the study noted that in zero-shot settings, larger is not always better; LLaMA-2-70B occasionally underperformed its 7B and 13B counterparts, suggesting that massive scale can sometimes introduce noise or inefficiencies for specific niche tasks like summarization.
Conclusion
The research confirms that fine-tuned compact models—specifically FLAN-T5-Large—can serve as highly effective, cost-efficient alternatives to massive LLMs for meeting summarization. Future work will focus on overcoming context window limitations through "chapterization" (segmenting long meetings) to ensure these Tiny Titans can handle even the most extensive transcripts without losing critical information.
Summary updated