A frame based spoken dialog system for home care

Daniele Falavigna, Toni Giorgino, Roberto GretterInterspeech 2005

TL;DR

This paper presents a frame-based spoken dialog system designed to monitor the health of hypertensive patients remotely via telephone. By refining dialog strategies and speech recognition grammars based on real-world usage, the system achieved significant improvements in accuracy and user experience, demonstrating the viability of such systems for chronic disease management.

What Problem Does This Paper Address?

Managing chronic diseases like hypertension requires consistent monitoring and patient engagement. Traditional methods often rely on manual check-ins or patient-initiated contact, which can be burdensome and prone to missed information. The authors address the challenge of creating an automated, accessible, and effective system for remote health monitoring that can collect clinical data, track lifestyle, and alert physicians when necessary. Specifically, they focus on the practical challenges of deploying such a system in a real-world clinical trial, including improving speech recognition accuracy, handling patient errors, and optimizing dialog flow for a non-expert user base.

Methods

The core of the system is a frame-based spoken dialog interface. This approach uses a "dialog description" document, manually defined by domain experts, which dictates the system's interaction flow. This description includes voice prompts and corresponding Automatic Speech Recognition (ASR) grammars for each piece of information to be acquired.

Key methodological contributions include:

  • Adaptive Dialog: The system maintains a "state vector" that represents the current call's progress and incorporates data from the patient's Electronic Health Record (EHR). This allows the system to adapt its behavior based on the patient's history and the ongoing interaction, creating a more personalized experience.
  • Dialog Refinements: The authors detail a process of iterative improvement based on initial evaluations. This involved:
  • Grammar Tuning: Enhancing grammar coverage and modifying number recognition to accept various formats (e.g., "sixty-seven" vs. "six seven").
  • Dialog Strategy Optimization: Replacing multiple confirmations with immediate, explicit confirmations to reduce user frustration.
  • Error Handling: Introducing "spelled-digit" modes for number entry to facilitate corrections.
  • Evaluation Framework: The paper outlines a comprehensive evaluation strategy, combining technical metrics (Word Accuracy, Semantic Accuracy) with clinical trial data and user feedback.

Datasets

The evaluation was conducted using data collected during a 12-month clinical trial involving 125 hypertensive patients.

  • Data Sources:
    • Call Logs: Generated by the computer telephony platform, capturing interaction metadata.
  • Clinical Information: Patient data stored in the Electronic Health Record (EHR).
  • Speech Data:
    • Recorded Dialogs: 541 dialogs were collected from real patients.
  • Transcribed Dialogs: A significant portion of these dialogs were orthographically transcribed for analysis.
  • Benchmark: The paper implicitly introduces a benchmark for spoken dialog systems in home healthcare, providing detailed statistics on performance metrics before and after dialog refinements.

Key Insights and Ideas

  • Real-world deployment is challenging: Even with well-designed systems, user errors, acoustic ambiguities (e.g., similar-sounding numbers), and usability issues are significant hurdles.
  • Manual refinement is crucial: While the system is adaptive, manual tuning of ASR grammars and dialog strategies based on real user data is essential for achieving high performance in practical applications.
  • Flexibility in input is key: Allowing patients to enter numbers in multiple ways (e.g., "sixty-seven" vs. "six seven") significantly improves usability and reduces errors.
  • Simplicity in confirmation: Replacing complex, multi-step confirmations with direct, single-step confirmations drastically improves user satisfaction and reduces confusion.
  • User expertise matters: Recognition rates improve as patients gain experience with the system over multiple dialogs.

Key Results

  • Improved Speech Recognition:
    • Grammar Coverage: Increased from 43 grammars to 36 after refinements, with coverage rates reaching up to 95.87%.
  • Word Accuracy (StrRR): Improved from 77.85% (feb04) to 85.78% (sep04) for the overall dataset. The "covered" subset (sentences expected to be recognized) saw even higher rates, reaching 91.56% in sep04-cov.
  • Semantic Accuracy: Reported at 88.10% for the sep04 dataset, indicating that the system correctly understood the meaning of the user's utterances.
  • Reduced User Frustration: The fraction of dialogs where patients hung up before the final salutation dropped remarkably after refinements.
  • Clinical Hint: A preliminary analysis suggested a statistically significant decrease (p < 0.10) in 24-hour average systolic blood pressure for patients using the system, though more data is needed for conclusive results.

Limitations and Future Work

  • Limitations:
    • The evaluation of clinical outcomes is preliminary due to a limited sample size for ABPM measurements.
  • The performance gap between male and female patients, though reduced, still exists.
  • The manual transcription of data for refinement is labor-intensive.
  • Future Work:
    • Further investigation into clinical outcomes with more data.
  • Exploring the use of additional knowledge sources (medical, clinical) to enhance the system's clinical effectiveness.
  • Developing more automated methods for grammar and dialog refinement.

Summary updated

BibTeX
@inproceedings{falavigna2005frame,
  author = {Daniele Falavigna and Toni Giorgino and Roberto Gretter},
  title = {{A frame based spoken dialog system for home care}},
  booktitle = {Interspeech},
  year = {2005},
}
Figure