Abstract
Instructional documents are rich sources of knowledge for completing various tasks, yet their unique challenges in conversational question answering (CQA) have not been thoroughly explored. Existing benchmarks have primarily focused on basic factual question-answering from single narrative documents, making them inadequate for assessing a model's ability to comprehend complex real-world instructional documents and provide accurate step-by-step guidance in daily life. To bridge this gap, we present InsCoQA, a novel benchmark tailored for evaluating large language models (LLMs) in the context of CQA with instructional documents. Sourced from extensive, encyclopedia-style instructional content, InsCoQA assesses models on their ability to retrieve, interpret, and accurately summarize procedural guidance from multiple documents, reflecting the intricate and multi-faceted nature of real-world instructional tasks. Additionally, to comprehensively assess state-of-the-art LLMs on the InsCoQA benchmark, we propose InsEval, an LLM-assisted evaluator that measures the integrity and accuracy of generated responses and procedural instructions.
Introduction: The Shift from Narrative to Instructional CQA
Most existing Conversational Question Answering (CQA) benchmarks, such as CoQA or QuAC, have become increasingly trivial for modern Large Language Models (LLMs). These datasets typically rely on single narrative documents—like news articles or stories—where questions require simple factual recall (e.g., "Who had a birthday?").
However, real-world user needs are significantly more complex. On social and instructional platforms like Xiaohongshu or Lemon8, users seek actionable guidance: how to repair a device, cook a meal tailored to a specific health condition, or configure software. These tasks present three unique challenges that traditional benchmarks fail to capture:
- Multi-document Synthesis: Answers are rarely contained in a single source; they must be aggregated from multiple instructional documents.
- Procedural Reasoning: The output must be a logical, step-by-step sequence rather than a simple paragraph.
- Conversational Context: Models must maintain coreference resolution (e.g., "In that case, what should I eat?") across multiple turns while referencing technical instructions.
The InsCoQA Benchmark
To address these limitations, researchers from USTC and Xiaohongshu introduced InsCoQA. This dataset was constructed using a rigorous pipeline starting with popular user queries. GPT-4 was employed to filter for high-quality, objective instructional queries, followed by the retrieval of multiple relevant documents for each query.
Dataset Characteristics
The resulting dataset comprises 13,959 conversational samples across 13 diverse domains, including Gourmet, Health & Fitness, Tech, and Home Decor. On average, each conversation references approximately 4.1 documents and culminates in 6.5 procedural steps. This structure forces models to synthesize information across disparate sources to provide a cohesive answer.
INSEVAL: A Task-Oriented Evaluation Framework
Traditional metrics like ROUGE-L are often insufficient for instructional tasks because they prioritize text matching over logical correctness. A model might use different phrasing but provide perfectly accurate steps. To solve this, the authors proposed INSEVAL, an LLM-assisted evaluation framework (using Qwen2-7B-Instruct or GPT-4) that focuses on two key metrics:
- Judge Score (1-10): An assessment of overall helpfulness and relevance.
- Task Completion Rate (TCR): A specific calculation of the percentage of correct procedural steps provided compared to the ground truth.
Key Insights and Experimental Results
The researchers benchmarked several state-of-the-art models, revealing a significant "Alignment Gap" between a model's ability to provide a conversational response and its ability to distill that response into accurate procedural steps.
- The Gold Standard: GPT-4 remains the leader, achieving a 91.37% Task Completion Rate.
- The "WizardLM Surprise": Among open-weight models, WizardLM-2-7B demonstrated exceptional performance. It achieved an 87.83% TCR, outperforming larger or more recent models like Llama-3.1-8B (82.98%). Interestingly, WizardLM-2-7B often scored lower on ROUGE-L, suggesting it is better at following the logic of instructions even when it deviates from the reference phrasing.
- Efficient Evaluation: The study found that a well-tuned 7B model (Qwen2) serves as a competent judge. While its absolute scores vary slightly from GPT-4, it maintains the same relative ranking of models, providing a cost-effective benchmarking solution.
Conclusion and Future Directions
InsCoQA highlights that "knowing that" (facts) is fundamentally different from "knowing how" (procedures). While current LLMs are proficient at the former, synthesizing conflicting or scattered instructions remains a challenge.
Future work in this domain should address the inherent biases of LLM-based evaluators and expand the benchmark to include "open-domain" instructional QA, where models must search the live web. Furthermore, as real-world instructions are often visual, incorporating multimodal documents (images and videos) is the next critical frontier for conversational instructional assistants.
Summary updated