TL;DR
This paper introduces SpeechLLMs, a novel approach that unifies speech and text processing for spoken language understanding (SLU) tasks like slot filling. By leveraging large language models (LLMs) adapted for speech, they achieve significant improvements in zero-shot generalization to unseen slot labels, paving the way for more adaptable and efficient conversational AI systems.
What Problem Does This Paper Address?
Modern conversational AI systems need to understand spoken language in complex, dynamic environments. Traditionally, this involves a multi-step pipeline: speech-to-text (ASR) followed by natural language understanding (NLU). This cascaded approach suffers from error propagation (ASR errors impacting NLU) and disjoint optimization. Furthermore, existing SLU systems are often limited to predefined domains and slot types, making them inflexible. The challenge is to create a unified system that can directly process speech and text, understand context, and generalize to new, unseen slot labels without extensive retraining. This paper tackles this by exploring SpeechLLMs for zero-shot slot filling, aiming to bridge the gap between current capabilities and an ideal, highly adaptable system.
Methods
The core of this work is the development and evaluation of SpeechLLMs, which tightly couple speech and text modalities within a decoder-only LLM architecture. The proposed architecture consists of three main components:
- Speech Encoder: Processes raw audio signals.
- Modality Adapter: Transforms speech features into a representation compatible with the LLM's text input space. The paper investigates various adapter architectures (CNN, Linear, MLP, Transformer) with increasing complexity.
- Pretrained Large Language Model (LLM): The backbone that handles both speech and text inputs to generate textual outputs.
Key methodological contributions include:
- Instruction-based Training: Using LLMs to generate diverse instructions for slot filling, enabling better generalization.
- Contextualized Prompting: Incorporating the entire conversation history into prompts for more robust slot extraction.
- Multistage Training: Employing strategies like fine-tuning the encoder and LLM separately before joint fine-tuning to improve stability and performance.
- Multitask Learning: Incorporating auxiliary tasks (like Automatic Speech Transcription - AST) to enhance multimodal alignment and prevent overfitting.
- Parameter-Efficient Fine-Tuning (PEFT): Utilizing techniques like QLoRA to efficiently fine-tune large models on limited resources.
Datasets
The paper introduces a new dataset and utilizes existing ones:
- CallCenter-A: A newly curated dataset of scripted call-center conversations across banking, telecommunication, insurance, and retail domains. It contains approximately 31K calls, 1M turns, and 2.1K hours of speech. This dataset was annotated using GPT-40 for slot labels and values.
- CallCenter-B: An evaluation dataset created from real call-center conversations, designed to test out-of-domain generalization. It includes subsets for in-domain (CallCenter-B ID) and out-of-domain (CallCenter-B OOD) slots.
- Internal AST Dataset: A subset of an English training corpus for production models, used for fine-tuning the Whisper ASR component.
- Spoken Alpaca Datasets (SIT & SQIT): Spoken versions of textual Alpaca instruction datasets, used as auxiliary tasks.
Key Insights and Ideas
- Modality Adapters Matter: The complexity and design of the modality adapter significantly impact performance. Simpler adapters might be more efficient but less effective than more complex ones like MLPs or Transformers.
- Modality Alignment is Crucial: Ensuring that speech and text inputs are aligned correctly for the LLM is paramount for effective multimodal understanding.
- Multistage Training Boosts Performance: A staged approach to training (e.g., pre-training components, then joint fine-tuning) leads to faster convergence and better overall performance compared to single-stage training.
- Multitask Data Enhances Robustness: Incorporating auxiliary tasks, even if not the primary focus, helps improve generalization and robustness, especially under constrained model capacity.
- Foundational SpeechLLMs Offer an Advantage: Starting with a pre-trained foundational SpeechLLM (like Qwen2-Audio) provides a significant performance boost but may offer less flexibility for task-specific adaptation compared to composite models.
- Zero-Shot is Key for Adaptability: The ability to generalize to unseen slot labels without explicit training is critical for dynamic conversational AI systems.
Key Results
- The proposed SpeechLLM approach significantly outperforms baseline systems, including a cascaded ASR+NLU pipeline and a simple SpeechLLM with a basic adapter.
- Multistage training strategies (especially Multistage-C) consistently improved performance on the CallCenter-A dataset, achieving F1 scores up to 0.7916.
- The models demonstrated varying degrees of robustness and generalization on out-of-domain datasets (CallCenter-B ID and OOD), with performance gaps highlighting the challenges of true multimodal integration.
- Fine-tuning a foundational SpeechLLM (Qwen2-Audio) yielded strong results, but the paper emphasizes the trade-off between performance and flexibility.
Limitations and Future Work
- Single Performance Estimates: Due to compute constraints, only a single performance estimate per model configuration was reported, making it difficult to assess statistical significance of small performance differences.
- Limited Validation: Human annotations were limited to a small subset, relying more on LLM-generated data.
- No Open Datasets: The study did not report results on public benchmarks due to the scale of the considered datasets. While the dataset is not distributable, reproducibility is ensured through detailed documentation.
- Model Landscape Coverage: The paper acknowledges that presenting results on only a few representative models may not cover the full diversity of current LLM capabilities.
- Future Work: The authors suggest continued scaling of models and data, refined training strategies, and further exploration of architectural components to improve robustness and generalization, particularly for out-of-domain scenarios and zero-shot capabilities. They also highlight the need to address the trade-offs between composite and foundational SpeechLLMs.
Summary updated