Speech-Based Web Navigation for Limited Mobility Users

Vasiliy Radostev, Serge Berger, Justin Tabrizi, Pasha Kamyshev, Hisami SuzukiInterspeech 2019

TL;DR

This paper introduces a novel browser extension that integrates voice assistant capabilities for web navigation, significantly improving task completion rates for users with limited mobility. By combining ASR, NLU, and a smart classifier to handle n-best ASR results, the system makes web browsing more accessible than traditional voice assistants alone.

What Problem Does This Paper Address?

While voice assistants have become adept at handling simple commands and answering questions, their application to complex tasks like web navigation has lagged. Traditional web navigation relies heavily on mouse, keyboard, or touch input, posing a significant barrier for users with limited mobility. Existing voice assistants often struggle with the nuances of web navigation, leading to frustration and task failure. This paper aims to bridge this gap by creating a system that seamlessly integrates voice-based web navigation into a browser, making it more accessible and efficient for everyone, especially those facing mobility challenges.

Methods

The core of the proposed system is a web browser extension that incorporates Automatic Speech Recognition (ASR), Text-to-Speech (TTS), and Natural Language Understanding (NLU) components. What's novel is how these components are integrated for web navigation. Instead of relying on a single ASR output, the system leverages the "n-best" results from the ASR component. A classifier, using universally consistent algorithms like SVM and AdaBoost, is employed to decide the best course of action based on these multiple ASR hypotheses. This approach aims to improve end-to-end accuracy by jointly optimizing ASR, NLU, and web navigation outcomes. The system's architecture is illustrated in Figure 1, showing the flow from speech input through the browser extension to the web browser.

Datasets

The paper describes an offline evaluation of a boosting classifier using 10K labeled data, split into 70% training and 25% testing data for a 10-fold cross-validation experiment. For the main experiment evaluating the proposed system, the authors created a test set of 30 pairs of websites (A) and search queries (B). This set was assembled from a random sample of web navigation usage data from users who consented to data collection and analysis by Microsoft.

Key Insights and Ideas

  • Beyond Simple Commands: The paper highlights that while voice assistants excel at basic tasks, their utility diminishes for more complex interactions like web navigation. This research pushes the boundaries of voice assistant applications.
  • N-Best ASR for Robustness: Relying on multiple ASR hypotheses (n-best results) and using a classifier to select the best interpretation is a key strategy to overcome speech recognition errors, a common pain point for voice assistant users.
  • Joint Optimization is Crucial: The authors emphasize the importance of jointly optimizing ASR, NLU, and the web navigation action for better overall performance, rather than treating them as independent modules.
  • Accessibility as a Primary Driver: The motivation for this work is strongly rooted in improving accessibility for users with limited mobility, demonstrating how technology can be adapted to overcome physical barriers.

Key Results

The experiments compared three systems: S1 (traditional browser with mouse/keyboard/touch), S2 (standard voice assistant like Cortana/Google Assistant), and S3 (the proposed speech-based web navigation system).

  • Traditional Browsing (S1): Achieved near-perfect task completion rates (100% on PC, 96% on mobile).
  • Standard Voice Assistants (S2): Showed significantly lower completion rates, especially on PC (7%), with mobile being slightly better (22%), suggesting voice assistants are more optimized for mobile environments.
  • Proposed System (S3): Dramatically improved task completion rates over S2, achieving 52% on PC and 48% on mobile. This demonstrates that the proposed system makes web navigation substantially more accessible for users who cannot rely on traditional input methods.

Limitations and Future Work

The authors acknowledge that while their system significantly boosts task completion compared to standard voice assistants (S2), it still falls short of the performance of traditional input methods (S1). They plan to continue improving the system by enhancing the performance of its core components (ASR, NLU) and by incorporating real user feedback. The paper also notes that the classifier used in the offline evaluation (SVM/AdaBoost) was not yet integrated into the main experiment but is slated for future use to further improve web navigation task performance. The system is currently integrated into the Microsoft Bing Search app for mobile and a PC version is upcoming.

Summary updated

BibTeX
@inproceedings{radostev2019speech,
  author = {Vasiliy Radostev and Serge Berger and Justin Tabrizi and Pasha Kamyshev and Hisami Suzuki},
  title = {{Speech-Based Web Navigation for Limited Mobility Users}},
  booktitle = {Interspeech},
  year = {2019},
}
Figure