TL;DR
This paper introduces SELMA, a unified Speech-Enabled Language Model for virtual assistants that integrates audio and text inputs to a single Large Language Model (LLM). SELMA simultaneously handles multiple crucial tasks like Automatic Speech Recognition (ASR), Voice Trigger (VT) detection, and Device-Directed Speech Detection (DDSD), significantly simplifying traditional multi-model pipelines and achieving competitive or improved performance.
What Problem Does This Paper Address?
Virtual assistants typically rely on complex, multi-stage pipelines to process user interactions. This often involves separate models for tasks like detecting a wake word (Voice Trigger detection), determining if speech is directed at the assistant (Device-Directed Speech Detection - DDSD), and transcribing speech (ASR). These pipelines can be inefficient and prone to errors, especially when dealing with noisy audio or when ASR transcripts are imperfect. Furthermore, DDSD often requires separate audio- and text-based processing, adding further complexity. This paper aims to address this by proposing a single, end-to-end multimodal model that can handle these tasks concurrently, streamlining the process and potentially improving robustness.
Methods
The core of SELMA is a multimodal LLM built upon the instruction-finetuned Qwen-Audio-Chat model. It comprises:
- Audio Encoder: A Transformer-based encoder that converts raw audio into a sequence of latent representations.
- Aggregator: This component (optional) reduces the length of the audio representations, often using mean pooling, to capture global audio patterns. This is beneficial for tasks less reliant on fine-grained temporal details.
- Gating Network: Another optional component that dynamically modulates the contribution of individual audio features to the final representation, allowing the model to adapt to different task requirements.
- LLM Backbone: A decoder-only LLM (based on Qwen) that processes both the audio representations and textual prompts to generate outputs for the various tasks.
A key innovation is the joint, end-to-end training of the audio encoder and the LLM. To achieve this efficiently, the authors employ Low-Rank Adaptation (LoRA), a parameter-efficient finetuning technique. LoRA attaches small, trainable adapter modules to the LLM and audio encoder, allowing them to specialize in the new tasks without modifying the entire pre-trained model. This significantly reduces the number of trainable parameters and improves inference efficiency. The model is trained using a multi-task objective with a specific prompt-based strategy to differentiate between tasks.
Datasets
The paper utilizes several datasets for training and evaluation:
- VT Detection: Data similar to [17] and [3], augmented with noise and room impulse responses. The training set contains ~7M queries.
- DDSD: Updated versions of datasets from [9], including ~237k queries and an additional ~3.8M text-only sentences for regularization.
- ASR: A mixture of in-house corpora, totaling ~232k training queries and ~10k validation queries.
- Auxiliary DA Classification: A randomly sampled subset of VT and DDSD training data, automatically annotated.
Crucially, the VT and DDSD corpora lack ground truth transcripts. To address this, an auxiliary model was used to generate ASR transcripts for these datasets, which then served as ground truth for the main SELMA model's training.
Key Insights and Ideas
- Unified Multimodal Processing: The most significant insight is that a single, end-to-end multimodal LLM can effectively handle multiple distinct tasks (ASR, VT, DDSD) that are typically managed by separate models. This drastically simplifies the traditional virtual assistant pipeline.
- End-to-End Training Benefits: Jointly training the audio encoder and LLM, rather than using frozen encoders, leads to better performance. This allows the audio representations to be optimized specifically for the downstream tasks.
- LoRA for Efficiency: Employing LoRA for finetuning is a practical and effective way to adapt large pre-trained models to new multimodal tasks without the computational cost of full finetuning.
- Importance of Sequence and Aggregation: The experiments show that retaining the full audio sequence representation alongside an aggregated global representation (mean pooling) is crucial for robust performance, especially for VT and DDSD. This suggests that both fine-grained temporal details and overall audio context are important.
- Prompting for Task Differentiation: The use of specific textual prompts effectively guides the single model to perform different tasks, demonstrating the LLM's flexibility and in-context learning capabilities.
Key Results
SELMA demonstrates strong performance across all tasks:
- VT Detection: Achieves a relative Equal-Error Rate (EER) improvement of 64% compared to baselines.
- DDSD: Achieves a relative EER improvement of 22% compared to baselines.
- ASR: Achieves word error rates (WER) close to dedicated baseline models.
- Model Component Analysis: "SELMA 1" (mean pooling + sequence concatenation) shows the best overall performance. Disabling pooling or sequence concatenation leads to degraded performance.
- Auxiliary Task Impact: Removing auxiliary tasks (text-only DDSD, DA) had a minor impact, suggesting the core tasks are well-learned. However, removing ASR as a standalone task led to a notable WER increase, highlighting the benefit of jointly training ASR.
Limitations and Future Work
- Limitations: The paper acknowledges that the VT and DDSD datasets lack ground truth transcripts, necessitating an auxiliary model for their generation. While SELMA performs well, the WER is not always on par with highly specialized ASR models.
- Future Work: The authors propose extending SELMA to integrate downstream Natural Language Understanding (NLU) components. The ultimate goal is to create a single, unified model that handles the entire user input processing pipeline, from raw audio to assistant response generation.
Summary updated