A user study to compare two conversational assistants designed for people with hearing impairments

Anja Virkkunen, Juri Lukkarila, Kalle Palomäki, Mikko KurimoNAACL 2019

TL;DR

This paper presents a user study comparing two conversational assistants for people with hearing impairments: a laptop-based system and a mobile augmented reality (AR) system. The study found that while both systems are useful, the mobile AR version was preferred, particularly for its ability to integrate lip-reading cues. The primary challenge remains the accuracy of automatic speech recognition (ASR) in real-world conversational settings.

What Problem Does This Paper Address?

Conversations can be exhausting and isolating for individuals with hearing loss. While hearing aids and implants help, they struggle in noisy environments, and many people find them uncomfortable or costly. Professional interpreters are an option but require advance booking and are expensive. Automatic Speech Recognition (ASR) offers a potential solution by providing near real-time transcriptions at a lower cost and with greater independence. However, current ASR systems face challenges with robustness in noisy environments, speaker identification, and capturing non-verbal cues like gestures, which are crucial for understanding in conversations. This paper addresses the need to understand user preferences for ASR-based conversational assistants, specifically comparing a traditional text-based approach with an augmented reality approach that aims to integrate visual cues.

Methods

The researchers developed two prototypes of a conversational assistant:

  • Laptop Version: A standalone system running ASR locally on a laptop. This served as a baseline for evaluating the core ASR concept.
  • Mobile AR Version: A system designed for a mobile device (iPad) that wirelessly connects to an ASR server. This version uses augmented reality to overlay transcriptions as speech bubbles next to the speaker's face, aiming to reduce the need for users to switch attention between the speaker and the text.

Both systems utilized the Kaldi Speech Recognition Toolkit for ASR. The mobile AR version's acoustic model was trained on a larger dataset (Finnish Parliament Speech Corpus) than the laptop version's (SPEECON corpus), leading to a lower word error rate (WER) on broadcast news data.

User tests were conducted with deaf and hard-of-hearing participants. They engaged in simulated noisy conversations and word-explanation tasks, comparing the systems with and without the assistant, or comparing the text-only view with the AR view. Feedback was collected through questionnaires and interviews.

Datasets

  • SPEECON corpus: Used for training the acoustic model of the laptop ASR system.
  • Finnish Parliament Speech Corpus: Used for training the acoustic model of the mobile AR ASR system. This corpus is noted as being significantly larger than SPEECON.
  • YLE (Finnish public broadcaster) broadcast news data: Used for evaluating the ASR models.

Key Insights and Ideas

  • AR enhances usability: The mobile AR version was generally preferred over the text-only view, primarily because it allowed users to combine transcriptions with lip-reading, which is a natural strategy for many individuals with hearing impairments.
  • Integration is key: The AR approach's success lies in its attempt to integrate visual cues (lip movements) with auditory information (transcriptions), reducing cognitive load.
  • ASR accuracy is paramount: Despite the benefits of AR, the accuracy of the ASR transcriptions was the most significant limiting factor for both systems. Errors in transcription directly impacted the perceived helpfulness of the assistants.
  • Context matters for system choice: While the AR view was preferred overall, participants noted that a text-only view might be more suitable for certain situations, like meetings, where direct eye contact is less critical.
  • Device preference is for mobility: Participants overwhelmingly favored mobile devices (smartphones, tablets) for conversational assistants, highlighting the need for portability and integration into daily life. Smart glasses, while a potential future platform, were seen as bulky and impractical by most.

Key Results

  • Overall Usefulness: A significant majority (83%) of participants indicated they would use a conversational assistant in their daily lives.
  • Preference for AR: 67% of participants preferred the AR view over a text-only view in the mobile system.
  • Transcription Speed vs. Accuracy: 67% of users preferred faster transcriptions even if they contained more mistakes, while 33% preferred slower but more accurate results.
  • Helpfulness Ratings: Both systems were rated favorably for helpfulness, with the AR version generally receiving slightly higher ratings, especially in the conversation task.
  • Speech Recognition Accuracy: This was identified as the biggest problem. While the mobile AR system had a lower WER (18.56% on YLE data) compared to the laptop system (29.7% on YLE data), errors still significantly impacted user experience.

Limitations and Future Work

  • ASR Accuracy: The paper explicitly states that ASR accuracy is the biggest issue needing improvement. This includes robustness in noisy environments and the lack of speaker diarization (identifying who is speaking).
  • Non-verbal Cues: While AR helps with lip-reading, capturing other paralinguistic features like tone and pitch remains a challenge.
  • Social Appropriateness: Some users expressed concern about the social appropriateness of pointing a device camera at a conversation partner.
  • Future Work: The authors suggest that speaker diarization would greatly improve the usefulness of the assistant, especially for group conversations. They also note that AR glasses could potentially solve the issue of direct eye contact but need to be unobtrusive and lightweight for wider adoption. The need for unobtrusive, wearable technology for seamless integration is also implied.

Summary updated

BibTeX
@inproceedings{virkkunen2019user,
  author = {Anja Virkkunen and Juri Lukkarila and Kalle Palomäki and Mikko Kurimo},
  title = {{A user study to compare two conversational assistants designed for people with hearing impairments}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2019},
}
Figure