Referring to Screen Texts with Voice Assistants

Shruti Bhargava, Anand Dhoot, Ing-marie Jonsson, Hoang Long Nguyen, Alkesh Patel, Hong Yu, Vincent RenkensACL 2023

TL;DR

Voice assistants are great, but they struggle to understand context on your phone screen. This paper introduces a new way for users to refer to on-screen text elements (like phone numbers or addresses) using natural language, enabling more intuitive interactions. They developed a dataset and a lightweight model to make this "screen referencing" a reality for voice assistants.

What Problem Does This Paper Address?

Voice assistants (VAs) have become ubiquitous, offering a convenient way to interact with our devices. However, their ability to understand user context is limited. Imagine you're looking at a webpage with multiple phone numbers and want to share one with a friend. The current VA experience forces you to either memorize the number and dictate it (prone to errors and unnatural) or manually copy-paste. This paper tackles the challenge of enabling VAs to understand natural language references to specific text elements on a user's screen, bridging the gap between user intent and VA comprehension.

Methods

The core of this work is the Screen Reference Resolver (SRR), a modular, attention-based network designed for efficient on-device inference. SRR leverages existing signals like request embeddings and UI text extracted via OCR. It comprises three key modules:

  • Category Module: Embeds entity categories (e.g., "phone number") and matches them with the request.
  • Location Module: Uses bounding box information of entities to understand spatial references.
  • Text Module: Employs string matching features (like word overlap) to compare the request text with the on-screen text, avoiding the computational overhead of embedding all screen texts.

The system is designed to be lightweight and privacy-preserving, crucial for on-device deployment. They also propose a Heuristic-based Baseline for quick prototyping, using a sequence of rules based on keywords, location, and text matching.

Datasets

The authors introduce ScreenRef, a new dataset designed to capture requests referring to actionable text on screen. ScreenRef has two collections:

  • Descriptive Data: Collected using screens with multiple similar entities (e.g., several phone numbers) to encourage challenging, disambiguating references.
  • Category-level Data: Collected without screens, focusing on unique, diverse requests for a given entity category (e.g., "call this number"). This collection is designed for scalability and efficiency.

Key Insights and Ideas

  • Naturalness over Literalness: Users prefer referring to screen elements using natural language ("Send that office number to Tim") rather than reciting the full text, especially for longer or complex entities like URLs.
  • The Power of Labels: In their user study, users frequently used labels or surrounding text to refer to entities, indicating a reliance on contextual clues beyond just the entity itself.
  • On-Device Efficiency is Key: For practical VA deployment, models need to be lightweight, low-latency, and privacy-preserving, driving the design of SRR.
  • Modularity for Flexibility: The modular architecture of SRR allows for flexibility, enabling different modules to be weighted more heavily based on the request type, and potentially allowing for skipping modules to save resources.

Key Results

  • SRR Significantly Outperforms Baseline: On the Descriptive data, SRR achieved a Top-1 Error of 14.2% compared to the heuristic baseline's 25.0%. For Category-level data, SRR achieved 1.1% Top-1 Error versus the heuristic's 6.5%.
  • Module Importance: An ablation study showed that each of SRR's modules (Category, Location, Text) is critical for achieving high performance, highlighting the need for multi-modal understanding.
  • User Preference for References: Their user study revealed that a majority of users (57%) preferred using references like "Send that office number to Tim" over repeating the full text.

Limitations and Future Work

  • Limited Entity Categories: The current work focuses on a few specific categories (phone numbers, addresses, emails, URLs, dates/times). Future work could expand to include a wider range of on-screen texts and visual elements.
  • No Image Context: The SRR model currently relies solely on extracted text and does not incorporate image context. Incorporating visual cues could improve coverage and understanding.
  • Upstream Dependency: The system's performance is bounded by the accuracy of upstream text extraction and entity detection.
  • Holistic Screen Understanding: While SRR considers entities, future work could explore a more holistic understanding of the entire screen's layout and content.

Summary updated

BibTeX
@inproceedings{bhargava2023referring,
  author = {Shruti Bhargava and Anand Dhoot and Ing-marie Jonsson and Hoang Long Nguyen and Alkesh Patel and Hong Yu and Vincent Renkens},
  title = {{Referring to Screen Texts with Voice Assistants}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
}

Loading related content...

Figure