Multi-turn Dialog System on Single-turn Data in Medical Domain

Nazib Sorathiya, Chuan-An Lin, Daniel Chen Daniel Xiong, Scott Zin, Yi Zhang, He Sarina Yang, Sharon Xiaolei HuangarXiv 2021 · arXiv

Introduction

In the medical field, the demand for accurate, interactive AI systems is high, yet the data required to train them is often restricted. While the internet contains vast amounts of health information, much of it is unverified. To address this, researchers from UC Santa Cruz and partner institutions have proposed a novel framework for building multi-turn medical chatbots using only "single-turn" data, such as verified FAQs and doctor-patient question-answer (QA) pairs. This approach bypasses the need for massive, privacy-sensitive conversational datasets by reformulating static records into interactive clarification tasks.

The Challenge of Medical Dialogue Data

Training a multi-turn dialog system typically requires extensive logs of real-world conversations. In medicine, these are difficult to acquire due to strict privacy regulations and the high cost of expert verification. However, there is an abundance of verified single-turn data from medical consulting sites and laboratory FAQs. The core research question is: Can we train a chatbot to be conversational using data that is inherently non-conversational?

Methodology

The researchers developed two distinct strategies to bridge the gap between static data and interactive dialogue:

1. Rule-Based Retrieval for FAQs

For structured FAQ data (e.g., from Lab Tests Online), the system employs a three-part pipeline. It matches a user's incomplete query to a known FAQ, tokenizes the original FAQ into specific "conditions" and "questions," and then generates a clarification question based on the condition missing from the user's initial input.

2. Probabilistic Modeling for Medical Dialogues

For more complex interactions, the team utilized a Naive Bayes model based on a "hidden symptom" assumption. They identified symptoms mentioned by both the patient and the doctor in a single-turn record and then "hid" one of those symptoms from the patient’s initial description. The model's task is to calculate the "criticalness" of various symptoms to determine which follow-up question would most effectively narrow down the diagnosis.

Figure 1: Illustration of reformulating a single-turn patient-doctor interaction into a multi-turn clarification dialogue.

Datasets and Knowledge Integration

The study utilized two primary sources:

  • Lab Tests Online: 2,529 FAQ pairs regarding laboratory tests, filtered to approximately 100 pairs suited for clarification tasks.
  • HealthcareMagic.com: A massive dataset of over 200,000 single-turn dialogue pairs. After filtering for records with clear, single diagnoses, 8,494 dialogues were used for training and evaluation.

To standardize the process, the system integrated a symptom-disease knowledge base containing 131 symptoms and 31 disease terms.

Key Results

The researchers found that reformulation effectively creates a synthetic multi-turn training environment.

  • FAQ Performance: The rule-based model achieved a 62% coverage rate in successfully generating clarification questions for incomplete user queries.
  • Symptom Prediction: Using the Naive Bayes approach on the HealthcareMagic dataset, the model’s ability to predict the exact "hidden" symptom (Recall@1) was modest at 0.0326. However, performance improved significantly when considering the top 10 predicted symptoms (Recall@10), reaching 0.2369.
Figure 3: Recall metrics for hidden symptom prediction, showing the increase in accuracy as the number of considered symptoms (k) grows.

Discussion and Future Work

While the study demonstrates the feasibility of using single-turn data, several limitations remain. The current system is largely restricted to two-turn interactions (User query -> System clarification), whereas real-world medical consultations often require multiple stages of questioning. Additionally, the probabilistic model is sensitive to "noise" in natural language, as doctors often mention symptoms for general advice rather than strictly for diagnosis.

Future research will focus on moving toward deep learning architectures to extract clarification questions directly from text and expanding the model to handle more complex, multi-stage conversations. This work represents a significant step toward making high-quality medical AI more accessible by leveraging existing, verified data sources.

Summary updated

Abstract

Recently there has been a huge interest in dialog systems. This interest has also been developed in the field of the medical domain where researchers are focusing on building a dialog system in the medical domain. This research is focused on the multi-turn dialog system trained on the multi-turn dialog data. It is difficult to gather a huge amount of multi-turn conversational data in the medical domain that is verified by professionals and can be trusted. However, there are several frequently asked questions (FAQs) or single-turn QA pairs that have information that is verified by the experts and can be used to build a multi-turn dialog system.
BibTeX
@misc{sorathiya2021multi,
  author = {Nazib Sorathiya and Chuan-An Lin and Daniel Chen Daniel Xiong and Scott Zin and Yi Zhang and He Sarina Yang and Sharon Xiaolei Huang},
  title = {{Multi-turn Dialog System on Single-turn Data in Medical Domain}},
  year = {2021},
  eprint = {2105.12887},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2105.12887},
}
Figure