Efficient Data Generation for Source-grounded Information-seeking Dialogs: A Use Case for Meeting Transcripts

Lotem Golany, Filippo Galgani, Maya Mamo, Nimrod Parasol, Omer Vandsburger, Nadav Bar, Ido DaganEMNLP 2024 · DOI

Introduction

Developing AI agents capable of navigating long, unstructured documents like meeting transcripts is a significant challenge in natural language processing. Such agents must not only answer complex queries but also provide accurate source attribution. This paper, presented at EMNLP 2024, introduces MISeD (Meeting Information Seeking Dialogs), a high-quality dataset created through a novel semi-synthetic pipeline. By combining the generative capabilities of Large Language Models (LLMs) with targeted human verification, the researchers demonstrate that semi-synthetic data can match the efficacy of traditional, manually-intensive collection methods while significantly reducing costs.

The Challenge of Meeting Transcripts

Training a "meeting assistant" to answer questions like "What was the final decision on the budget?" requires large-scale, multi-turn dialog data grounded in source text. Traditionally, this data is gathered using the Wizard-of-Oz (WOZ) method, where two humans simulate a user-agent interaction. However, WOZ is slow, expensive, and difficult to scale. Meeting transcripts add further complexity: they are often lengthy, filled with disfluencies ("ums" and "ahs"), and feature overlapping speakers. While LLMs excel at text generation, they frequently struggle with precise attribution—citing the exact segment in a long transcript that supports a given answer.

Methodology: A Semi-Automatic Pipeline

To address these bottlenecks, the authors propose a workflow that leverages the strengths of both AI and human annotators.

The semi-automatic pipeline for generating MISeD dialogs, involving iterative LLM generation and human verification.

The process consists of four main stages:

  1. Iterative LLM Generation: Using Gemini Pro, the system generates dialog turns sequentially. A "query prompt" utilizes templates to ensure diverse user questions, while a "response prompt" directs the LLM to answer based on the transcript.
  2. Human-in-the-Loop Verification: Professional annotators review the generated turns to filter out hallucinations or nonsensical queries.
  3. Manual Attribution: Recognizing that LLMs are currently unreliable at precise citations in long contexts, humans manually identify the specific transcript segments (attribution spans) that support the generated answers.
  4. Model Fine-tuning: The resulting data is used to fine-tune models like LongT5 and Gemini Pro to evaluate the impact of synthetic-human hybrid data on performance.

The MISeD Dataset

The primary contribution of this work is the MISeD dataset. It is derived from 225 meetings within the QMSum corpus, encompassing various domains such as academic (AMI, ICSI) and parliamentary meetings.

Statistical overview of the MISeD dataset, including query types and average lengths.

As shown in the statistics, the dataset contains over 4,000 query-response pairs with a diverse mix of general, specific, and yes/no questions. Approximately 30% of the queries are designed to be unanswerable based on the transcript, testing the model's ability to recognize information gaps and avoid hallucinations.

Key Insights and Results

The study yielded several critical findings regarding data efficiency and model performance:

  • Efficiency Gains: The semi-automatic MISeD process was 1.5x faster than the manual WOZ process, requiring 105 minutes per dialog compared to 161 minutes.
  • Quality and Consistency: Human evaluations revealed that MISeD-generated responses were rated as "better" or "equally good" compared to fully human-written responses 57% of the time. LLM-generated queries often proved more consistent and structured than unconstrained human inputs.
  • Performance Boosts: Fine-tuning a 3-billion parameter LongT5 model on MISeD allowed it to approach the performance of much larger models like Gemini Ultra in zero-shot settings. Specifically, fine-tuning Gemini Pro on MISeD improved its semantic correctness (BLEURT score) from 0.48 to 0.52.
  • Attribution Accuracy: While zero-shot models performed poorly at citing sources, fine-tuning on MISeD boosted the Attribution F1 score for LongT5 from near-zero to 0.66.

Conclusion and Future Directions

The MISeD project demonstrates that a hybrid approach to data generation can produce high-quality, grounded dialog datasets more efficiently than traditional methods. While human intervention is still required for precise attribution, the authors anticipate that improvements in LLM long-context capabilities will eventually allow for further automation. Future work will explore increasing query diversity through LLM-driven "brainstorming" and integrating Retrieval-Augmented Generation (RAG) to handle even longer meeting contexts without truncation.

Summary updated

BibTeX
@inproceedings{golany2024efficient,
  author = {Lotem Golany and Filippo Galgani and Maya Mamo and Nimrod Parasol and Omer Vandsburger and Nadav Bar and Ido Dagan},
  title = {{Efficient Data Generation for Source-grounded Information-seeking Dialogs: A Use Case for Meeting Transcripts}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2024},
  doi = {10.18653/v1/2024.findings-emnlp.106},
}
Figure