Large Language Models Need Holistically Thought in Medical Conversational QA

Yixuan Weng, Bin Li, Fei Xia, Minjun Zhu, Bin Sun, Shizhu He, Kang Liu, Jun ZhaoarXiv 2023 · DOI · arXiv

Abstract

Medical Conversational Question Answering (CQA) requires a complex synthesis of broad diagnostic possibilities and deep patient-specific data. Standard prompting techniques often fail to capture this duality. We introduce Holistically Thought (HoT), a zero-shot prompting framework that guides Large Language Models (LLMs) to think both "diffusedly" (exploring multiple diagnostic paths) and "focusedly" (summarizing patient history). Our evaluations across three major medical datasets demonstrate that HoT significantly outperforms direct generation and standard Chain-of-Thought prompting, achieving results that approach human-level professional standards.

Introduction

In clinical practice, a physician does not follow a simple linear logic path. Instead, they engage in holistic reasoning: weighing various potential diagnoses (breadth) while simultaneously grounding their decisions in the patient's specific medical history and symptoms (depth). While Large Language Models (LLMs) like GPT-3 have shown remarkable general reasoning capabilities, they often struggle with the specialized complexity of healthcare.

Direct responses from LLMs can be shallow or factually incorrect. Even Chain-of-Thought (CoT) prompting, while effective for mathematical or logical tasks, often fails in medicine because it typically follows a single reasoning trajectory rather than exploring the diffused possibilities inherent in a differential diagnosis. To address this, we propose the Holistically Thought (HoT) framework.

Methodology

The HoT method decomposes the medical reasoning process into three distinct, zero-shot stages designed to mimic clinical cognition:

  1. Diffused Thinking (Breadth): The LLM uses "diversified decoding" to generate multiple, diverse potential responses. This forces the model to explore various "thought contents" and potential diagnostic angles for the patient's reported issues.
  2. Focused Thinking (Depth): The model extracts and generates a concise "medical record" from the dialogue history. This focuses the model on essential patient-specific information, including chief complaints, medical history, and previous examinations.
  3. Response Generation: Finally, the LLM synthesizes the original dialogue history, the diffused thoughts, and the focused medical record to produce a single, high-quality, professional response.
The three-stage workflow of the Holistically Thought (HoT) framework.

Experimental Evaluation

Datasets

We evaluated HoT across three diverse benchmark datasets:

  • MedDialog: A large-scale English dataset with over 250,000 consultations.
  • COVID: A specialized English dataset focused on COVID-19 consultations.
  • CMDD: A Chinese medical conversational dataset covering internal medicine, pediatrics, and oncology.

Quantitative Results

HoT demonstrated substantial improvements over state-of-the-art (SOTA) baselines. On the COVID dataset, GPT-3’s BLEU-2 score increased by 14.1 points compared to direct generation. In many scenarios, HoT doubled the performance of standard Chain-of-Thought prompting.

Performance comparison of HoT vs. Few-shot prompting across BLEU-2, Meteor, and Nist-2 metrics on COVID and CMDD datasets.

Qualitative Insights

Manual evaluations by medical professionals confirmed the metric gains. HoT-generated answers were rated significantly higher for correctness, professionalism, and considerateness. On a scale of 1-2, HoT consistently scored near 0.9 for correctness across all tested datasets.

Key findings include:

  • Breadth + Depth > Linear Logic: Combining diffused and focused thinking outperformed using either component individually.
  • Prompt Simplicity: Simple, direct prompts (e.g., "#1 - Doctor:") often performed better than overly complex, high-level instructions that could restrict the model's multifaceted thinking.
  • Zero-Shot Efficacy: As a zero-shot paradigm, HoT often outperformed traditional few-shot prompting, suggesting that guiding internal reasoning is more effective than providing external examples in complex domains.

Limitations and Future Work

While effective, HoT introduces higher computational costs due to multiple decoding passes. Additionally, models occasionally exhibit reduced fluency by "copying" previous text when generating longer responses. Future research will explore automatic prompt engineering to optimize the HoT stages and further reduce human intervention in system setup.

Conclusion

Holistically Thought (HoT) represents a significant step forward in medical CQA. By structuring LLM reasoning to mirror the breadth and depth of human clinical thought, we can leverage general-purpose models to provide more accurate, professional, and considerate medical advice without the need for specialized fine-tuning.

Summary updated

Abstract

The medical conversational question answering (CQA) system aims at providing a series of professional medical services to improve the efficiency of medical care. Despite the success of large language models (LLMs) in complex reasoning tasks in various fields, such as mathematics, logic, and commonsense QA, they still need to improve with the increased complexity and specialization of the medical field. This is because medical CQA tasks require not only strong medical reasoning, but also the ability to think broadly and deeply. In this paper, to address these challenges in medical CQA tasks that need to be considered and understood in many aspects, we propose the Holistically Thought (HoT) method, which is designed to guide the LLMs to perform the diffused and focused thinking for generating high-quality medical responses. The proposed HoT method has been evaluated through automated and manual assessments in three different medical CQA datasets containing the English and Chinese languages. The extensive experimental results show that our method can produce more correctness, professional, and considerate answers than several state-of-the-art (SOTA) methods, manifesting its effectiveness. Our code in https://github.com/WENGSYX/HoT.
BibTeX
@misc{weng2023large,
  author = {Yixuan Weng and Bin Li and Fei Xia and Minjun Zhu and Bin Sun and Shizhu He and Kang Liu and Jun Zhao},
  title = {{Large Language Models Need Holistically Thought in Medical Conversational QA}},
  year = {2023},
  eprint = {2305.05410},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2305.05410},
}
Figure