Introduction
Developing AI agents capable of navigating long, unstructured documents like meeting transcripts is a significant challenge in natural language processing. Such agents must not only answer complex queries but also provide accurate source attribution. This paper, presented at EMNLP 2024, introduces MISeD (Meeting Information Seeking Dialogs), a high-quality dataset created through a novel semi-synthetic pipeline. By combining the generative capabilities of Large Language Models (LLMs) with targeted human verification, the researchers demonstrate that semi-synthetic data can match the efficacy of traditional, manually-intensive collection methods while significantly reducing costs.
The Challenge of Meeting Transcripts
Training a "meeting assistant" to answer questions like "What was the final decision on the budget?" requires large-scale, multi-turn dialog data grounded in source text. Traditionally, this data is gathered using the Wizard-of-Oz (WOZ) method, where two humans simulate a user-agent interaction. However, WOZ is slow, expensive, and difficult to scale. Meeting transcripts add further complexity: they are often lengthy, filled with disfluencies ("ums" and "ahs"), and feature overlapping speakers. While LLMs excel at text generation, they frequently struggle with precise attribution—citing the exact segment in a long transcript that supports a given answer.
Methodology: A Semi-Automatic Pipeline
To address these bottlenecks, the authors propose a workflow that leverages the strengths of both AI and human annotators.
The process consists of four main stages:
- Iterative LLM Generation: Using Gemini Pro, the system generates dialog turns sequentially. A "query prompt" utilizes templates to ensure diverse user questions, while a "response prompt" directs the LLM to answer based on the transcript.
- Human-in-the-Loop Verification: Professional annotators review the generated turns to filter out hallucinations or nonsensical queries.
- Manual Attribution: Recognizing that LLMs are currently unreliable at precise citations in long contexts, humans manually identify the specific transcript segments (attribution spans) that support the generated answers.
- Model Fine-tuning: The resulting data is used to fine-tune models like LongT5 and Gemini Pro to evaluate the impact of synthetic-human hybrid data on performance.
The MISeD Dataset
The primary contribution of this work is the MISeD dataset. It is derived from 225 meetings within the QMSum corpus, encompassing various domains such as academic (AMI, ICSI) and parliamentary meetings.
As shown in the statistics, the dataset contains over 4,000 query-response pairs with a diverse mix of general, specific, and yes/no questions. Approximately 30% of the queries are designed to be unanswerable based on the transcript, testing the model's ability to recognize information gaps and avoid hallucinations.
Key Insights and Results
The study yielded several critical findings regarding data efficiency and model performance:
- Efficiency Gains: The semi-automatic MISeD process was 1.5x faster than the manual WOZ process, requiring 105 minutes per dialog compared to 161 minutes.
- Quality and Consistency: Human evaluations revealed that MISeD-generated responses were rated as "better" or "equally good" compared to fully human-written responses 57% of the time. LLM-generated queries often proved more consistent and structured than unconstrained human inputs.
- Performance Boosts: Fine-tuning a 3-billion parameter LongT5 model on MISeD allowed it to approach the performance of much larger models like Gemini Ultra in zero-shot settings. Specifically, fine-tuning Gemini Pro on MISeD improved its semantic correctness (BLEURT score) from 0.48 to 0.52.
- Attribution Accuracy: While zero-shot models performed poorly at citing sources, fine-tuning on MISeD boosted the Attribution F1 score for LongT5 from near-zero to 0.66.
Conclusion and Future Directions
The MISeD project demonstrates that a hybrid approach to data generation can produce high-quality, grounded dialog datasets more efficiently than traditional methods. While human intervention is still required for precise attribution, the authors anticipate that improvements in LLM long-context capabilities will eventually allow for further automation. Future work will explore increasing query diversity through LLM-driven "brainstorming" and integrating Retrieval-Augmented Generation (RAG) to handle even longer meeting contexts without truncation.
Summary updated