Gathering Information and Engaging the User ComBot: A Task-Based, Serendipitous Dialog Model for Patient-Doctor Interactions

Anna Liednikova, Philippe Jolivet, Alexandre Durand-Salmon, Claire GardentNAACL 2021

TL;DR

This paper introduces ComBot, a novel ensemble dialog model designed for clinical studies to gather both structured patient data and serendipitous health information. By combining task-oriented, follow-up, and empathy bots, ComBot aims to improve user engagement and data collection efficiency compared to single-bot approaches.

What Problem Does This Paper Address?

Clinical studies often require patients to provide not only answers to a predefined questionnaire but also to share any other health-related information that might be relevant to the study's outcome. This dual goal presents a challenge: how to efficiently collect both structured data and spontaneous, potentially valuable insights while keeping the patient engaged in the conversation. Traditional dialog models are often either too rigid, focusing solely on predefined tasks, or too open-ended, failing to capture specific medical information. This paper addresses the need for a dialog system that can navigate this complexity, ensuring both comprehensive data collection and a positive user experience in a medical context.

Methods

The core of ComBot is an ensemble of three distinct bots:

  • MEDBOT: A task-based retrieval model that uses the pre-trained ConveRT model to query the MedTree Corpus. Its primary function is to collect predefined data points by asking questions from a structured dialog tree.
  • FOLLOWUPBOT: This bot is designed to generate health-related questions that naturally follow from the user's input, even if they are not part of the predefined dialog tree. It uses the ConveRT model to retrieve questions from the newly created HealthBoard dataset, enabling the collection of serendipitous information and smoother dialog transitions.
  • EMPATHYBOT: A generative model (based on the Blender model) focused on user engagement. Its role is to provide empathetic and socially driven feedback, reinforcing patient interaction and making the dialog feel more human.

The ensemble model combines these three bots. When generating a response, each bot provides a candidate. The system then uses the ConveRT encoder to calculate the similarity between the current dialog context and each candidate response, selecting the one with the highest similarity score.

Datasets

  • MedTree Corpus: Developed to train task-based, information-seeking health bots. It's derived from a physician-created dialog tree covering domains like sleep, mood, and anxiety. The paper uses a modified version of the "INIT" subset for retrieval.
  • HealthBoard Dataset: A new dataset created by the authors for training the FOLLOWUPBOT. It consists of (statement, question) pairs extracted from the Healthboard online forum, specifically focusing on health-related conversations. This dataset is crucial for enabling the bot to generate relevant follow-up questions.
  • Reddit, ConvAI2, WoW, EmpaDial, BSD: These datasets are mentioned as pre-training or fine-tuning corpora for the underlying models (ConveRT and Blender) used by the bots.

Key Insights and Ideas

  • The Power of Ensemble: Combining specialized bots (task-oriented, follow-up, and empathy) is more effective than using a single bot for complex dialog tasks like clinical data collection.
  • Serendipity is Key: The FOLLOWUPBOT's ability to generate contextually relevant, unplanned questions is crucial for uncovering valuable, unstructured health information that questionnaires might miss.
  • Engagement Matters: The EMPATHYBOT plays a vital role in keeping users engaged, which in turn can lead to more open and informative interactions.
  • User Expectations: Users tend to ask more questions and provide more detailed information when they perceive the system as more capable and less restricted (like ComBot), compared to a system with limited scope (like MEDBOT).

Key Results

  • Improved Informativity: ComBot consistently outperformed MEDBOT in informativity metrics (Slots, InfoGain), indicating it collects a wider range of information in fewer turns.
  • Efficiency: ComBot achieved shorter dialogs (ConvLen) compared to MED+EMPATHYBOT, suggesting a more efficient way to gather information.
  • Coherence: ComBot showed the highest dialog coherence (CoSim) among the tested configurations, suggesting that follow-up questions contribute to smoother conversational flow.
  • Human Preference: In comparative human evaluations (Acute-Eval), users preferred ComBot, judging it as more knowledgeable, interesting, human-like, and better for long conversations.

Limitations and Future Work

  • Negation Handling: The model sometimes struggles to recognize negation, leading to continued discussion of topics the user has declared irrelevant.
  • Ending Conversations: Determining the optimal point to end a conversation remains a challenge, as users may feel compelled to continue even when ready to finish.
  • Emotional Impact: The paper suggests future work could explore sentiment analysis to better maintain positive interactions and improve negation handling.
  • Providing Solutions: For users who express distress or seek advice, a knowledge-based agent could be integrated to provide facts, advice, or highlight connections between mentioned information.
  • User Stress: The authors acknowledge that interactions might highlight health issues, increasing user stress. Future work could focus on providing solutions or advice to help users manage their health.

Summary updated

BibTeX
@inproceedings{liednikova2021gathering,
  author = {Anna Liednikova and Philippe Jolivet and Alexandre Durand-Salmon and Claire Gardent},
  title = {{Gathering Information and Engaging the User ComBot: A Task-Based, Serendipitous Dialog Model for Patient-Doctor Interactions}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2021},
}
Figure