Evaluation Dimensions for Assessing Question Answer Systems for Lay Users: The Case of DiseaseGuru

Prakash Chandra Sukhwal, Atreyi Kankanhalli, Vaibhav RajanAAAI 2023 · DOI

Abstract

Question answer (QA) systems can serve as vital tools to address lay users’ information needs in healthcare. While QA systems have the potential to lessen information overload and provide quality answers to users, it is important to holistically evaluate their performance. Here we propose multiple dimensions for this purpose comprising lexical similarity, semantic similarity, absence of contradictions and readability of responses. We then use the dimensions to evaluate DiseaseGuru, a generative large language model-based chronic disease QA system we developed that integrates knowledge graph technology to provide quality responses to lay users. The results are presented comparing it with three benchmark algorithms across the different dimensions. We also propose metrics for lay users and medical professionals for a future field study to evaluate the system.

Introduction

Lay users seeking health information online often face a "triple threat": information overload, outdated or incorrect advice, and dense medical jargon. While Large Language Models (LLMs) like ChatGPT can generate natural-sounding answers, they are prone to "hallucinations" (factual errors) and do not always prioritize readability for non-experts.

Current evaluation metrics for healthcare Question-Answering (QA) systems are often too narrow, focusing primarily on lexical similarity (word-for-word overlap). This paper addresses the need for a more robust, four-dimensional evaluation framework to ensure medical AI is accurate, easy to read, and internally consistent. We introduce DiseaseGuru, a hybrid system designed specifically to meet these needs for chronic disease information.

Methodology

System Architecture

DiseaseGuru utilizes a multi-component architecture to ensure factual grounding:

  • Vector Database (VDB): Stores an extensive library of medical documents.
  • Fine-tuned Sentence Transformer: Adapted for the medical domain to retrieve relevant contexts.
  • Knowledge Graph (KG): Acts as a factual anchor to select the best possible answer from the LLM’s potential outputs, reducing the risk of hallucinated errors.

Evaluation Framework

We propose a four-dimensional evaluation suite:

  1. Lexical Similarity: Using ROUGE to measure word-for-word overlap.
  2. Semantic Similarity: Using MedSTS and BERTScore to determine if the meaning is correct, even if different terminology is used (e.g., "myocardial infarction" vs. "heart attack").
  3. Readability: Using the Flesch-Kincaid (FK) score to ensure the text is appropriate for a general audience.
  4. Contradictions: A novel approach using pair-wise sentence entailment to check if any part of the generated answer contradicts the ground truth.

Experimental Setup

The evaluation focused on the top 10 most common chronic conditions (e.g., Alzheimer’s, Diabetes, Asthma, Cancer), using 75 test questions from the MedQuAD dataset. Answers generated by the Google Search Engine (SE) served as the ground truth "gold standard." DiseaseGuru was compared against ChatGPT-3.5 and two other specialized healthcare QA architectures.

Results and Discussion

DiseaseGuru demonstrated superior performance across almost every metric. As shown in the comparative results table, DiseaseGuru significantly outperformed ChatGPT-3.5 in lexical accuracy (ROUGE 0.40 vs 0.20) and semantic meaning (MedSTS 3.03 vs 2.53).

Comparative performance of DiseaseGuru against benchmark models across four evaluation dimensions.

Readability and Factuality

A key insight from this study is that readability is a critical metric, not an afterthought. Using Kernel Density Estimate plots, we analyzed the distribution of FK scores. DiseaseGuru achieved a higher distributional overlap with Google SE (78.9%) compared to ChatGPT (73.7%), indicating that DiseaseGuru produces more consistently accessible text.

FK Score distribution overlap comparing DiseaseGuru (A) and ChatGPT (B) against Google Search Engine ground truth.

Furthermore, the integration of a Knowledge Graph provided a "fact-checking" layer that pure LLMs lack. While some benchmarks produced up to 3.5 contradictions per answer, DiseaseGuru maintained a mean contradiction score of zero, highlighting the hybrid advantage for safety-critical medical information.

Conclusion and Future Work

The proposed four-dimensional framework provides a more holistic view of QA system performance than traditional metrics. While automated metrics show that DiseaseGuru excels in accuracy and readability, they are not a total replacement for human judgment. Future work will involve a field study where lay users and medical professionals will manually rate the system on completeness, relevance, and informativeness to further validate these findings in a real-world setting.

Summary updated

Abstract

Question answer (QA) systems can serve as vital tools to address lay users’ information needs in healthcare. While QA systems have the potential to lessen information overload and provide quality answers to users, it is important to holistically evaluate their performance. Here we propose multiple dimensions for this purpose comprising lexical similarity, semantic similarity, absence of contradictions and readability of responses. We then use the dimensions to evaluate DiseaseGuru, a generative large language model-based chronic disease QA system we developed that integrates knowledge graph technology to provide quality responses to lay users. The results are presented comparing it with three benchmark algorithms across the different dimensions. We also propose metrics for lay users and medical professionals for a future field study to evaluate the system.
BibTeX
@inproceedings{sukhwal2023evaluation,
  author = {Prakash Chandra Sukhwal and Atreyi Kankanhalli and Vaibhav Rajan},
  title = {{Evaluation Dimensions for Assessing Question Answer Systems for Lay Users: The Case of DiseaseGuru}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2023},
  doi = {10.1609/aaaiss.v1i1.27484},
}
Figure