A Dialogue System for Assessing Activities of Daily Living: Improving Consistency with Grounded Knowledge

Zhecheng Sheng, Raymond Finzel, Michael Lucke, Sheena Dufresne, Maria Gini, Serguei PakhomovACL 2023

TL;DR

This paper introduces a novel dialogue system designed to improve the consistency of assessments for Activities of Daily Living (ADLs). By grounding responses in a knowledge base derived from real historical assessments and leveraging large language models, the system aims to provide a more reliable and reproducible training tool for assessors.

What Problem Does This Paper Address?

In healthcare, assessing a person's ability to perform Activities of Daily Living (ADLs) is crucial for determining their need for support. This assessment process, often conducted through interviews, faces challenges with consistency, especially when multiple assessors with varying expertise are involved. Novice assessors may lack the real-world experience to conduct these interviews effectively. Furthermore, ensuring factual consistency in AI-powered dialogue systems, particularly in sensitive domains like healthcare, is a significant concern. This paper tackles these issues by developing a dialogue system that simulates these assessment interviews, providing a consistent and grounded training environment for assessors.

Methods

The core of the system lies in its two main modules: Natural Language Understanding (NLU) and Natural Language Generation (NLG). For NLU, a query classification model is employed to map user queries to specific ADL domains, improving the system's ability to retrieve relevant information. This classification is performed using transformer-based models like DeBERTa, which outperform simpler methods.

For NLG, the system leverages large language models (LLMs), specifically fine-tuned versions of LLaMA. To ensure factual consistency, the system grounds its responses in a knowledge base constructed from de-identified historical ADL assessments. This knowledge base contains synthetic profiles detailing individuals' functioning levels across 18 ADL domains. The system uses a heuristic to decide whether to generate a response directly from the LLM or to retrieve information from the knowledge base. Parameter-Efficient Fine-Tuning (PEFT) methods, particularly Low-Rank Adaptation (LoRA), are used to adapt LLMs to the specific ADL domain without the prohibitive cost of full model fine-tuning.

Datasets

  • Synthetic Dialogues Dataset: A newly created dataset comprising approximately 2,885 dialogues collected from a survey of 1,700 certified assessors. This dataset includes dialogue turns, ADL domains, and the interviewee's ability level. It was used to fine-tune the query classification model.
  • Historical Assessments Dataset: A collection of 10,000 de-identified historical ADL assessments from the Minnesota Department of Human Services. This dataset was used to create synthetic profiles and form the knowledge base for the dialogue system.
  • Stanford Alpaca Dataset: A public instruction-following dataset used for initial training of the LoRA adapter.
  • ADL Conversation Specific Dataset: A dataset created by manually annotating assessor notes into natural conversations, used for fine-tuning the LLM for ADL-specific dialogue. This dataset contains 6,123 question/answer pairs and examples of short conversations.

Key Insights and Ideas

  • Knowledge Grounding is Crucial for Factual Consistency: Even powerful LLMs can "hallucinate" or forget details. Grounding responses in a curated knowledge base, even one composed of natural language facts, significantly improves factual consistency without necessarily sacrificing conversational quality.
  • Synthetic Profiles Mimic Real-World Data: Creating synthetic profiles from de-identified historical assessments allows for the generation of diverse and realistic simulated patient interactions, which is vital for training assessors.
  • Parameter-Efficient Fine-Tuning (PEFT) is Key: Full fine-tuning of large LLMs is computationally expensive. PEFT methods like LoRA offer a viable alternative for adapting these models to specialized domains like ADL assessments, achieving comparable performance.
  • Domain Imbalance is a Challenge: The performance of classification models can be significantly affected by the imbalance of data across different ADL domains, highlighting the need for careful data augmentation or specialized handling.
  • Human Evaluation Remains Essential: Despite advancements in automatic metrics, human evaluation is still critical for assessing the nuanced qualities of dialogue systems, particularly sensibleness, specificity, and factual consistency in complex conversational scenarios.

Key Results

  • Query Classification: Transformer-based models, particularly DeBERTa, significantly outperformed a simple logistic regression model in domain classification, achieving the best performance across multiple metrics. Interestingly, augmented data (paraphrased utterances) did not consistently improve performance, suggesting potential noise introduction.
  • Response Generation: The fine-tuned LLaMA 7B model combined with the knowledge base achieved the highest scores for Sensibleness (4.92) and Specificity (4.33) in human evaluations. While the zero-shot Vicuna 13B model scored higher on Specificity (5.00), the knowledge-grounded LLaMA model demonstrated a better balance and significantly fewer contradictions (Knowledge: 4, History: 1 vs. Vicuna's 5 and 2 respectively). The authors noted that LLMs can provide a more pleasurable chat experience in open-domain conversations, but grounding is essential for factual accuracy in specialized domains.

Limitations and Future Work

  • Limited Evaluators and Dialogues: The human evaluation was conducted with a small number of evaluators and a limited set of evaluated dialogues, which restricts the generalizability of the findings.
  • Need for Robust Evaluation Frameworks: Future work should focus on developing more robust and replicable evaluation frameworks, especially for longer and more complex conversations, and incorporating larger numbers of human raters.
  • Exploring Larger LLMs and Prompt Engineering: Investigating larger LLMs and more sophisticated prompt engineering techniques could further enhance performance.
  • Threshold and Parameter Testing: Further testing of thresholds and parameters in the knowledge-grounding system is needed to optimize the decision-making process between generative and retrieval-based responses.
  • Expanding the Dataset: The authors plan to expand the dataset by adding more profiles and addressing any factual inconsistencies introduced by human annotators.

Summary updated

BibTeX
@inproceedings{sheng2023dialogue,
  author = {Zhecheng Sheng and Raymond Finzel and Michael Lucke and Sheena Dufresne and Maria Gini and Serguei Pakhomov},
  title = {{A Dialogue System for Assessing Activities of Daily Living: Improving Consistency with Grounded Knowledge}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
}
Figure