Structured Chain-of-Thought Prompting for Few-Shot Generation of Content-Grounded QA Conversations

Md. Arafat Sultan, Jatin Ganhotra, Ramón Fernandez Astudillo

Abstract

We introduce a structured chain-of-thought (SCoT) prompting approach to generating content-grounded multi-turn question-answer conversations using a pre-trained large language model (LLM). At the core of our proposal is a structured breakdown of the complex task into a number of states in a state machine, so that actions corresponding to various subtasks, e.g., content reading and utterance generation, can be executed in their own dedicated states. Each state leverages a unique set of resources including prompts and (optionally) additional tools to augment the generation process. Our experimental results show that SCoT prompting with designated states for hallucination mitigation increases agent faithfulness to grounding documents by up to 16.8%. When used as training data, our open-domain conversations synthesized from only 6 Wikipedia-based seed demonstrations train strong conversational QA agents; in out-of-domain evaluation, for example, we observe improvements of up to 13.9% over target domain gold data when the latter is augmented with our generated examples.

Introduction

Large Language Models (LLMs) have demonstrated remarkable capabilities in text generation, yet they remain prone to "hallucinations"—the generation of factually incorrect or ungrounded information. This issue is particularly acute in closed-domain grounded Question Answering (QA), where a model must answer questions based strictly on a provided document. While standard Chain-of-Thought (CoT) prompting encourages models to generate intermediate reasoning steps, it often falls short in multi-turn conversations where the model may conflate internal knowledge with the provided text or fail to recognize when a document does not contain an answer.

This paper introduces Structured Chain-of-Thought (SCoT) prompting. By modeling the conversation generation process as a state machine, SCoT decouples the "reading" (comprehension and evidence selection) from the "writing" (response generation). This modular approach significantly improves faithfulness and allows for the creation of high-quality synthetic datasets from minimal human-labeled seeds.

The SCoT State Machine

The fundamental innovation of SCoT is the decomposition of a single generation task into four distinct, sequential states. This structure prevents the model from rushing to a response before verifying the evidence.

  1. User Utterance Generation (uu): The model generates a plausible user question based on the document and the preceding conversation history.
  2. Question Answerability Classification (ac): A critical gatekeeping step where the model determines if the grounding document actually contains the information needed to answer the question. If the answer is missing, the agent is programmed to acknowledge the limitation rather than hallucinating a response.
  3. Answer Sentence Selection (ss): The model identifies the specific sentences within the document that serve as the evidence for the answer.
  4. Agent Utterance Generation (au): The final response is generated, strictly constrained by the evidence selected in the previous state.

By isolating these tasks, SCoT allows for "LM augmentation." For instance, while a large model like Falcon-40B might handle general generation, a specialized, instruction-tuned model like FLAN-UL2-20B can be employed for the classification (ac) and selection (ss) states, where precision is more critical than creative fluency.

Experimental Setup and Datasets

The researchers evaluated SCoT using a few-shot approach, starting with only 6 hand-written Wikipedia-based conversations (comprising 32 turns) as seed demonstrations. From these seeds, they generated synthetic datasets of 1,000 and 10,000 conversations.

Performance was benchmarked against several key datasets:

  • DoQA: Focused on domain-specific FAQs in categories like Cooking, Travel, and Movies.
  • QuAC (Question Answering in Context): A standard large-scale dataset for multi-turn, document-grounded QA.

Key Results

1. Mitigation of Hallucinations

SCoT prompting increased agent faithfulness to grounding documents by up to 16.8% compared to standard prompting methods. By forcing the model to explicitly classify answerability and select evidence before speaking, the system effectively decoupled the intent to respond from the content of the response.

2. Synthetic Data Utility

One of the most significant findings was the efficacy of SCoT-generated synthetic data. When used for Supervised Fine-Tuning (SFT), augmenting existing human-labeled "gold" data with SCoT synthetic data improved performance by 10–14% on out-of-domain tasks. In some cases, agents trained on synthetic data derived from just 6 seeds outperformed those trained on larger, noisier human datasets, suggesting that the structural consistency of SCoT data is highly beneficial for model training.

3. Zero-Shot Robustness

Even without fine-tuning, using SCoT as a few-shot prompting strategy significantly improved the ability of base models (such as Llama-2-13B) to correctly handle unanswerable questions, a common failure point for standard LLMs.

Discussion and Future Work

The success of SCoT highlights the value of modularity in LLM workflows. By treating the LLM as a component within a state machine rather than a monolithic black box, developers can better control for accuracy and grounding.

However, limitations remain. Human evaluation of multi-turn conversations is expensive, leading the authors to rely on "LLM-as-a-judge" metrics. Furthermore, while the current work focuses on single-document grounding, future iterations of SCoT aim to address multi-document scenarios and more complex queries where answers are distributed across disparate sections of text. Expanding the diversity of the initial seed data is also expected to further enhance the robustness of the synthetic outputs.

Summary updated

Abstract

We introduce a structured chain-of-thought (SCoT) prompting approach to generating content-grounded multi-turn question-answer conversations using a pre-trained large language model (LLM). At the core of our proposal is a structured breakdown of the complex task into a number of states in a state machine, so that actions corresponding to various subtasks, e.g., content reading and utterance generation, can be executed in their own dedicated states. Each state leverages a unique set of resources including prompts and (optionally) additional tools to augment the generation process. Our experimental results show that SCoT prompting with designated states for hallucination mitigation increases agent faithfulness to grounding documents by up to 16.8%. When used as training data, our open-domain conversations synthesized from only 6 Wikipedia-based seed demonstrations train strong conversational QA agents; in out-of-domain evaluation, for example, we observe improvements of up to 13.9% over target domain gold data when the latter is augmented with our generated examples.
BibTeX
@inproceedings{sultan2024structured,
  author = {Md. Arafat Sultan and Jatin Ganhotra and Ramón Fernandez Astudillo},
  title = {{Structured Chain-of-Thought Prompting for Few-Shot Generation of Content-Grounded QA Conversations}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2024},
  url = {https://openreview.net/forum?id=i9NSMJNCL2},
}
Figure