Introduction
Spoken Language Understanding (SLU) serves as the core intelligence of modern voice assistants, responsible for converting raw audio signals into structured semantic data. This process typically involves two key tasks: Intent Classification (IC), which identifies the user's goal (e.g., "Play Music"), and Slot Filling (SF), which extracts specific parameters (e.g., Artist: "Jay Chou"). As Large Language Models (LLMs) and Large Audio Language Models (LALMs) redefine the state of the art, researchers have introduced MAC-SLU, a new Chinese benchmark designed to push SLU into this new era by focusing on the complex, multi-intent environments of automotive cabins.
The Challenge: Beyond Simple Commands
Current SLU research faces two primary hurdles that MAC-SLU aims to overcome:
- Dataset Obsolescence: Classic benchmarks like ATIS and SNIPS are largely considered "solved," with modern models frequently exceeding 95% accuracy. These datasets primarily feature single, simple intents, which fail to capture the nuance of natural human-car interaction, where users often issue compound commands like "Navigate home and turn on the AC."
- Lack of Unified Benchmarking: While researchers are increasingly experimenting with LLMs and LALMs for SLU, the field lacks standardized prompts, formats, and evaluation methodologies. This fragmentation makes direct comparisons between different modeling paradigms difficult.
Introducing the MAC-SLU Dataset
To address these gaps, the researchers developed MAC-SLU, a comprehensive Chinese multi-intent dataset specifically tailored for automotive environments. The dataset comprises over 20,000 commands spanning 8 distinct domains, including Car Control, Map, and Music.
As shown in the comparison above, MAC-SLU significantly exceeds existing benchmarks like SLURP in complexity, featuring 81 intents and 192 slots. Crucially, it is built for multi-intent queries, containing sentences with up to five independent intents. The audio component was generated using the CosyVoice-2 TTS engine, utilizing diverse speaker embeddings to ensure acoustic variety.
Methodology: Evaluating SLU Paradigms
The study evaluates three distinct architectural approaches to SLU:
- NLU (Text-only): Transcribed text is fed into LLMs (e.g., Qwen3) for semantic parsing.
- Pipeline (ASR + NLU): An Automatic Speech Recognition (ASR) model like Whisper transcribes the audio before passing it to an LLM.
- End-to-End (E2E) SLU: LALMs (e.g., Qwen2.5-Omni, GPT-4o-Audio) process raw audio directly to generate semantic structures, potentially avoiding the "telephone game" effect where ASR errors propagate through the system.
These models were tested using both In-Context Learning (ICL)—providing examples within the prompt—and Supervised Fine-Tuning (SFT) via LoRA (Low-Rank Adaptation).
Results and Analysis
The experimental results highlight a significant performance gap between general-purpose prompting and domain-specific training. While LALMs show surprising potential in zero-shot slot filling when prompts are carefully engineered, their overall accuracy (correctly identifying both intent and all slots) remains below 15% without fine-tuning.
As the data indicates, Supervised Fine-Tuning is essential for high-precision applications. For instance, the Qwen2.5-Omni-7B model achieved an impressive 91.24% intent accuracy after fine-tuning. Furthermore, the E2E approach proved its viability by outperforming similarly sized pipeline systems, particularly in scenarios where ASR error rates were high. In the zero-shot arena, closed-source models like GPT-4o-Audio and Gemini-2.5-Flash continue to lead open-source alternatives in overall accuracy.
Conclusion and Future Directions
MAC-SLU establishes a new frontier for automotive voice interaction, revealing that while modern generative models are powerful, they require specialized tuning to meet the rigorous demands of cabin systems. The authors note that current evaluation metrics based on exact string matching may be too rigid for generative models, which often produce semantically correct but lexically different answers. Future work will focus on developing semantically-aligned evaluation metrics and improving the in-context learning capabilities of LALMs to reduce the reliance on expensive fine-tuning.
Summary updated