Improving Device Directedness Classification of Utterances With Semantic Lexical Features

Kellen Gillespie, Ioannis C. Konstantakopoulos, Xingzhi Guo, Vishal Thanvantri Vasudevan, Abhinav SethyICASSP 2020 · DOI

TL;DR

This paper tackles the challenge of accurately identifying when a spoken utterance is directed at a personal assistant, especially in "follow-up" modes where wake words are omitted. They propose a novel approach combining semantic lexical features with acoustic signals, achieving a significant reduction in error rate compared to acoustic-only methods. Furthermore, they demonstrate the effectiveness of transfer learning and semi-supervised learning to boost performance without extensive manual labeling.

What Problem Does This Paper Address?

Modern personal assistants like Alexa, Google Home, and Siri are increasingly incorporating "follow-up" modes. These modes allow for more natural, conversational interactions by enabling users to issue consecutive commands without repeating the wake word (e.g., "Alexa"). While this enhances user experience, it introduces a critical challenge: distinguishing between genuine device-directed commands and ambient, non-directed speech. Accurately classifying these utterances is crucial to prevent the assistant from responding inappropriately to background noise, media, or unrelated conversations. Existing methods often rely heavily on acoustic features, but this paper argues for the complementary power of semantic lexical information.

Methods

The core of this work lies in developing a directedness classifier that fuses multiple feature types. The proposed model leverages:

  • Semantic Lexical Features: These are derived from pre-trained word embeddings (fastText wiki news word vectors) to capture the meaning of words within an utterance. They also introduce a mechanism to handle out-of-vocabulary (OOV) tokens, which are common in personal assistant commands.
  • Acoustic Features: Specifically, token-level ASR (Automatic Speech Recognition) confidence scores are used as a lightweight acoustic signal.
  • Contextual Features: The model also incorporates information from the previous utterance in the conversation, recognizing that context can be vital for disambiguating current utterances.

The authors explore several model architectures:

  • AVG-DNN: A baseline feed-forward network that uses an average of token embeddings to represent the utterance.
  • LSTM: A recurrent neural network (RNN) that processes sequential information, expected to better handle word order and grammatical structure.
  • LSTM+Attention: An enhancement to the LSTM model that uses an attention mechanism to softly select relevant parts of the sequence for generating the final sentence embedding.

To address data labeling costs, they investigate:

  • Transfer Learning: Pre-training models on a large dataset of first-turn interactions (including false wake words) before fine-tuning on the follow-up dataset.
  • Semi-Supervised Learning: Employing a self-teaching methodology where a model trained on labeled data is used to label a larger unlabeled dataset, iteratively improving its performance. They also explore a co-training approach, combining scores from their proposed model and an acoustic-only baseline for more robust labeling.

Datasets

  • Follow-up Dataset: The primary dataset used for training and evaluation consists of approximately 240K utterances from personal assistant follow-up interactions, with ground truth labels for device-directed (DD) and non-device-directed (NDD) speech. This dataset is split into train (200K), dev (20K), and test (20K) partitions. The class distribution is imbalanced, with DD speech being significantly more frequent (roughly 5:1).
  • Pre-training Dataset: A large (2M utterances) ASR false-wake dataset is used for transfer learning. This dataset contains first-turn device-directed and non-device-directed speech.
  • Semi-Supervised Dataset: A dataset of 500K unlabeled utterances is used for semi-supervised learning experiments.

Key Insights and Ideas

  • Lexical Semantics Matter: The paper strongly advocates for the inclusion of semantic lexical features, demonstrating that they significantly improve directedness classification beyond acoustic-only approaches.
  • Context is King (Sometimes): While previous turn information is less critical than current turn text or acoustic features, it still provides a boost, especially for contextually ambiguous utterances.
  • LSTM Architecture Benefits: Recurrent models like LSTMs are better suited for this task than simple averaging models, likely due to their ability to capture sequential dependencies and grammatical structure, which helps in rejecting unstructured non-directed speech.
  • Attention Enhances LSTMs: The attention mechanism further refines the LSTM's performance by allowing it to focus on the most informative parts of the utterance.
  • Transfer and Semi-Supervised Learning are Powerful: These techniques offer substantial improvements in accuracy without the need for extensive manual data labeling, making them highly practical for real-world deployment.

Key Results

  • The proposed LSTM+Attention model with combined features (current utterance text, previous utterance text, and ASR confidence scores) achieved a 14% relative reduction in Equal Error Rate (EER) compared to the acoustic-only baseline model [5].
  • The token-level ASR confidence scores were found to be the most important feature, followed closely by the current turn text.
  • Transfer learning, when applied to the LSTM+Attention model, further reduced EER by 14% relative compared to the same model trained without transfer learning.
  • Semi-supervised learning, after 20 passes, improved test EER by 5% relative compared to the model trained solely on labeled data.

Limitations and Future Work

  • Contextual Ambiguity: The paper acknowledges that some utterances are inherently ambiguous and difficult to classify even with context.
  • Model Architectures: The authors plan to explore more advanced model architectures, including CNN-LSTMs, FLSTMs, and Transformer-style models.
  • Semi-Supervised Learning: Future work will involve experimenting with larger unlabeled datasets, more advanced graph-based techniques for exploration, and incorporating contextual features and attention into semi-supervised training.
  • Feature Fusion: While they combined features, further research into optimal fusion strategies could yield additional gains.

Summary updated

BibTeX
@inproceedings{gillespie2020improving,
  author = {Kellen Gillespie and Ioannis C. Konstantakopoulos and Xingzhi Guo and Vishal Thanvantri Vasudevan and Abhinav Sethy},
  title = {{Improving Device Directedness Classification of Utterances With Semantic Lexical Features}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2020},
  doi = {10.1109/icassp40776.2020.9054304},
}
Figure