STEER: Semantic Turn Extension-Expansion Recognition for Voice Assistants

Leon Zhang, Jiarui Lu, Joel Ruben Antony Moniz, Aditya Kulkarni, Dhivya Piraviperumal, Tien Dung Tran, Nick Tzou, Hong YuEMNLP 2023

TL;DR

This paper introduces STEER, a novel model for detecting "steering" in voice assistant conversations – when a user tries to clarify or redirect a previous command. They also propose STEER+, which enhances this by incorporating semantic parse trees, leading to significantly improved accuracy and a better user experience by reducing conversational friction.

What Problem Does This Paper Address?

Voice assistants are great, but they often struggle with "steering" – a common human conversational pattern where a user follows up a command with a clarification or modification. For example, if you ask a voice assistant to "play the song 'Bohemian Rhapsody'," and it plays the wrong version, you might then say, "No, the Queen version." This follow-up is a steering attempt. Current voice assistants often fail to recognize these steering attempts, forcing users to repeat their entire original request, which is frustrating and disrupts the natural flow of conversation. This leads to a poor user experience. The challenge is building a system that can reliably detect these steering intents, especially given that steering requests are relatively rare and difficult to simulate for training data.

Methods

The researchers propose two models:

  • STEER: This model is a transformer-based architecture that takes the text transcriptions of two consecutive turns (the original command and the follow-up) as input. It uses token, positional, and turn encodings to understand the relationship between the turns and predict whether the second turn is a steering attempt.
  • STEER+: This enhanced model builds upon STEER by incorporating an additional feature: a linearized Semantic Parse Tree (SPT) derived from the first turn. SPTs provide a structured representation of the command's intent, targets, and entities. This extra semantic context helps the model better understand out-of-vocabulary words and named entities, which are often crucial for steering. The SPT encoding is integrated as an additional token to the original query embedding.

A key innovation is their data sampling strategy. To overcome the "cold-start problem" (where poor steering support leads to a lack of steering data), they developed heuristic rules to automatically sample "opt-in" usage data. For positive (steering) examples, they identify reiterations where the second turn is a prefix of the first and synthetically generate a steering follow-up. For negative (non-steering) examples, they sample natural, self-contained follow-up requests. This allows them to create a large, unsupervised training dataset.

Datasets

The primary dataset used for training and evaluation was created using the authors' novel heuristic-based data sampling strategy. This strategy generated approximately four million samples, which were split into training, validation, and testing sets.

Additionally, they conducted a human-graded evaluation on over 800 real-world steering examples to assess the models' zero-shot performance in practical scenarios.

Key Insights and Ideas

  • Semantic Parse Trees are Powerful: Incorporating SPTs (STEER+) significantly boosts performance, especially in domains rich with entities (like messaging or social conversations). This highlights the value of structured semantic information beyond raw text.
  • Heuristic Data Sampling Works: The proposed heuristic approach for generating training data is effective, enabling the creation of a large, high-quality dataset without manual annotation, which is crucial for rare phenomena like steering.
  • Steering Reduces User Friction: The ability to detect steering directly translates to a better user experience by saving users from repeating themselves, leading to more natural and efficient interactions.
  • Conversational Naturalness: By handling steering, voice assistants can better accommodate human conversational patterns, including pauses for thought or clarification, making interactions feel more fluid and less robotic.

Key Results

  • High Accuracy: STEER achieved a macro accuracy of 95.99% on their heuristically sampled data. STEER+ improved this to 96.44%.
  • Real-World Performance: On a human-graded dataset, STEER achieved 90.71% accuracy, while STEER+ achieved a statistically significant improvement to 91.20%.
  • User Friction Reduction: STEER saves users an average of 3.96 words (58.06% of the query) from repetition. STEER+ further improves this to 4.09 words (58.64%).
  • Domain-Specific Gains: STEER+ showed particularly strong improvements in domains like messaging, social conversation, and images, where named entities are prevalent.

Limitations and Future Work

  • Heuristic Data Proxy: While effective, the positive training data is synthetically generated based on heuristics. The authors acknowledge this and validate their approach with real-world data.
  • SPT Encoding: The paper mentions that more advanced techniques like Tree-LSTMs or Tree Transformers could be explored for encoding the SPT, potentially leading to further improvements.
  • Model Size and Latency: While STEER and STEER+ are relatively comparable in size and training time, the paper notes that large language models, while powerful, can be prohibitively expensive for low-power devices, which is a common setting for voice assistants. Future work could focus on optimizing these models for such constraints.

Summary updated

BibTeX
@inproceedings{zhang2023steer,
  author = {Leon Zhang and Jiarui Lu and Joel Ruben Antony Moniz and Aditya Kulkarni and Dhivya Piraviperumal and Tien Dung Tran and Nick Tzou and Hong Yu},
  title = {{STEER: Semantic Turn Extension-Expansion Recognition for Voice Assistants}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2023},
}
Figure