How Accurate Are LLMs at Multi-Question Answering on Conversational Transcripts?

Xiliang Zhu, Shi Zong, David RossouwEMNLP 2025

TL;DR

This paper investigates the effectiveness of Large Language Models (LLMs) in answering multiple questions simultaneously from lengthy conversational transcripts, a common task in customer support. They find that while proprietary models like GPT-4o perform well, fine-tuned smaller open-source models can achieve comparable or even superior accuracy, highlighting their potential for cost-effective deployment.

What Problem Does This Paper Address?

Customer support centers generate vast amounts of conversational data (call transcripts). Analyzing these transcripts to answer specific questions is crucial for quality assurance, agent coaching, and identifying key customer issues. A significant challenge arises when multiple questions need to be answered from a single, often very long, transcript. Processing each question independently leads to high computational costs and latency. This paper tackles the problem of efficiently answering multiple questions from a single conversational context using LLMs, aiming to optimize inference without sacrificing accuracy.

Methods

The core of the paper's methodology revolves around adapting and evaluating "batch prompting" for multi-question answering on conversational transcripts. Instead of sending individual questions to an LLM, multiple questions are bundled together in a single prompt along with the transcript.

Key aspects of their approach include:

  • Structured Output: Responses are designed to be in a JSON format, including a binary "Judgment" (yes/no) and a "Navigation" component. The Navigation component provides an utterance index from the transcript that supports the judgment, enhancing explainability and mitigating hallucination.
  • Batch Prompting Adaptation: They adapt the batch prompting paradigm, which has shown promise for LLM APIs, to the specific domain of long, multi-turn conversational transcripts.
  • Zero-Shot and Fine-Tuning: They evaluate both proprietary and public LLMs in a zero-shot setting (without task-specific training) and also explore fine-tuning smaller public models on their specific task.
  • Varied Group Sizes: Experiments are conducted with varying "group sizes" (N), representing the number of questions bundled in a single prompt, to understand performance degradation as the query load increases.
  • Stochastic Training: For fine-tuning, they introduce variability in the number of question-answer pairs per training instance to improve model robustness and reduce undesirable output patterns.

Datasets

The paper utilizes a dataset of 2,800 conversation transcripts generated by their in-house Automatic Speech Recognition (ASR) engine from various contact centers.

  • Transcript Length: They provide statistics on transcript lengths, noting that over 25% exceed 2,700 tokens, with 5% reaching up to 6,500 tokens.
  • Annotation: Due to the cost and impracticality of large-scale human annotation for long transcripts, they used GPT-4o-generated labels as a reference, after validating strong agreement with human annotators for smaller group sizes (N < 20).
  • Train/Dev/Test Splits: The dataset is split into training (2,000 transcripts), testing (500 transcripts), and development (300 transcripts) sets. For evaluation, test and development sets are constructed with specific group sizes (N ∈ {10, 20, 30, 40, 50}).

Key Insights and Ideas

  • Fine-tuning is Key for Public LLMs: While proprietary models like GPT-4o perform well out-of-the-box, fine-tuning smaller public LLMs (e.g., Llama-3.1-8B, Qwen2.5-7B) significantly boosts their performance, making them competitive.
  • Smaller Models Can Compete: The study demonstrates that fine-tuned public LLMs with as few as 8 billion parameters can achieve accuracy comparable to, or even surpassing, larger proprietary models like GPT-4o, especially when a larger number of questions are grouped. This is a significant finding for cost-effective deployment.
  • Performance Degradation with Group Size: All models, including GPT-4o, show a decline in performance (both Judgment accuracy and Navigation F1/MAE) as the number of questions (group size N) in a single prompt increases. This highlights the inherent challenge of processing more information within a single inference pass.
  • Navigation Component Improves Trust: The inclusion of a "Navigation" component (supporting utterance index) is crucial for explainability and allows for easier human review, helping to detect and mitigate hallucinations.
  • JSON Decoding Challenges: Public LLMs, particularly in their instruction-tuned versions, can struggle with generating correctly formatted JSON outputs, though fine-tuning helps mitigate this issue.

Key Results

  • GPT-4o as a Strong Baseline: GPT-4o achieved the best overall performance across most metrics, especially for smaller group sizes (N=5 or 10), with Judgment accuracy around 0.95 and Navigation accuracy around 0.90.
  • Fine-tuned Public LLMs Surpass Proprietary Models: For Judgment accuracy, fine-tuned Qwen2.5-7B and Llama3.1-8B models surpassed GPT-40 when the group size N was 30 or more. For instance, Llama3.1-8B-10 achieved a Judgment accuracy of 0.87 with N=40, comparable to GPT-40's 0.85.
  • Performance Drop with Increased Queries: As group size N increased from 10 to 50, Judgment accuracy for GPT-40 dropped from ~0.90 to ~0.85, and Navigation F1 dropped from ~0.71 to ~0.63. Similar trends were observed for other models.
  • JSON Decode Error Reduction: Fine-tuning significantly reduced JSON decoding errors for public LLMs, bringing them closer to the near-zero error rates of proprietary models.

Limitations and Future Work

  • Limited Model Scope: The study did not benchmark some of the latest strong reasoning models (e.g., DeepSeek R1, QwQ, OpenAI O1) due to practical constraints.
  • Yes/No Questions Only: The experiments were limited to Yes/No type questions. Future work should explore other question types to validate the findings more broadly.
  • Annotation Cost: Obtaining high-quality human annotations for long transcripts and numerous questions is challenging and costly. While GPT-4o labels were validated, a full-scale human annotation job would provide more accurate benchmarking.
  • Future Directions: The authors suggest that their findings demonstrate the viability of fine-tuning smaller models for multi-question answering on long contexts, paving the way for cost-effective deployment. Future work could explore more sophisticated batching strategies or model architectures to further mitigate performance degradation with increasing query loads.

Summary updated

BibTeX
@inproceedings{zhu2025how,
  author = {Xiliang Zhu and Shi Zong and David Rossouw},
  title = {{How Accurate Are LLMs at Multi-Question Answering on Conversational Transcripts?}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
}
Figure