TL;DR
This paper introduces a multimodal approach to device-directed speech detection, enabling virtual assistants to understand when they are being addressed without requiring a specific trigger phrase. By combining audio, ASR (Automatic Speech Recognition) decoder signals, and text from LLMs (Large Language Models), the system achieves significant improvements in accuracy over unimodal methods.
What Problem Does This Paper Address?
Virtual assistants typically rely on a "wake word" or trigger phrase to distinguish commands directed at them from background conversations. This paper tackles the challenge of device-directed speech detection, which aims to identify when a user is addressing the assistant without a preceding trigger phrase. This is a more difficult problem because the system must infer intent from the speech content and context alone. Existing methods often combine various features like acoustics and lexical information, but this work explores how Large Language Models (LLMs) can be leveraged to integrate these modalities more effectively for this task.
Methods
The core of the proposed system is a decoder-only LLM (specifically GPT-2 variants) that acts as the central decision-maker. The system takes three types of input:
- Acoustic Features: Extracted from audio using pre-trained models like Whisper or CLAP. These models are chosen for their ability to generalize across various acoustic conditions.
- ASR Decoder Signals: These are supplementary signals from an ASR system, including the 1-best hypothesis (the most likely transcription), and utterance-level metrics like acoustic cost and graph cost. These signals provide richer information than just the text transcription alone.
- Text (1-best hypothesis): The standard transcription of the speech.
To integrate these diverse inputs, the system uses two mapping networks (M1 and M2). These networks transform the audio and ASR decoder features into a format compatible with the LLM's token embedding space, effectively creating "prefix tokens." The LLM is then trained to predict whether the utterance is device-directed based on this combined multimodal input. The paper also explores parameter-efficient fine-tuning using LoRA (Low-Rank Adaptation), which is crucial for adapting large LLMs without retraining all parameters.
Datasets
- Training Data: A balanced, anonymized dataset of approximately 40k device-directed and 40k non-directed utterances. A significant portion (21%) of the device-directed examples in this set do not start with a trigger phrase. An additional text corpus of ~3 million ASR-transcribed near-field speech signals was also used for text-only training and to simulate modality dropout.
- Evaluation Data: A combined set of ~14k device-directed and ~23k non-directed utterances from two in-house datasets. This evaluation set also includes triggerless device-directed examples.
Key Insights and Ideas
- Multimodality is Key: Combining audio, ASR decoder signals, and text significantly outperforms any single modality. This highlights the complementary nature of these information sources.
- LLMs as Integrators: LLMs are powerful tools for integrating diverse modalities due to their strong language understanding and ability to handle long contexts.
- ASR Decoder Signals Add Value: Beyond just the transcribed text, metrics from the ASR decoder (like costs) provide valuable signals for distinguishing directed speech.
- Whisper vs. CLAP Trade-offs: Whisper, trained on a vast amount of ASR data, initially provides better audio features for this task. However, CLAP, trained on audio captioning, becomes more effective when all modalities are combined, suggesting it captures different, complementary acoustic aspects.
- Parameter-Efficient Fine-tuning Matters: LoRA adaptation, especially with larger LLMs, offers a promising path for achieving high performance without the computational cost of full fine-tuning.
Key Results
The paper reports significant improvements in Equal Error Rate (EER), a standard metric for detection tasks:
- The best multimodal system (MM6, using GPT-2, Whisper, text, audio, and DS) achieved an EER of 7.95%.
- Using CLAP as the audio encoder with the same multimodal setup (MM6) resulted in an EER of 7.45%.
- These multimodal systems showed relative EER improvements of up to 39% over text-only models and up to 61% over audio-only models.
- When using LoRA with a larger GPT-2 1.5B model (MM6.3), EERs as low as 6.53% were achieved, demonstrating the effectiveness of parameter-efficient fine-tuning.
- The multimodal models consistently outperformed unimodal baselines across the entire detection error trade-off (DET) curve.
Limitations and Future Work
- LLM Fine-tuning: Full fine-tuning of LLMs can be computationally expensive and may reduce their general multi-task capabilities. The paper explores LoRA as a more practical alternative.
- Future Directions: The authors suggest extending the approach to incorporate other tasks relevant to virtual assistants, such as audio captioning and acoustic scene classification. They also plan to explore longer contexts, potentially by including features from previous interactions.
Summary updated