MAC-SLU: Multi-Intent Automotive Cabin Spoken Language Understanding Benchmark

Yuezhang Peng, Chonghao Cai, Ziang Liu, Shuai Fan, Sheng Jiang, Hua Xu, Yuxin Liu, Qiguang Chen, Kele Xu, Yao Li, Sheng Wang, Libo Qin, Xie ChenarXiv 2025 · DOI · arXiv

Introduction

Spoken Language Understanding (SLU) serves as the core intelligence of modern voice assistants, responsible for converting raw audio signals into structured semantic data. This process typically involves two key tasks: Intent Classification (IC), which identifies the user's goal (e.g., "Play Music"), and Slot Filling (SF), which extracts specific parameters (e.g., Artist: "Jay Chou"). As Large Language Models (LLMs) and Large Audio Language Models (LALMs) redefine the state of the art, researchers have introduced MAC-SLU, a new Chinese benchmark designed to push SLU into this new era by focusing on the complex, multi-intent environments of automotive cabins.

The Challenge: Beyond Simple Commands

Current SLU research faces two primary hurdles that MAC-SLU aims to overcome:

  1. Dataset Obsolescence: Classic benchmarks like ATIS and SNIPS are largely considered "solved," with modern models frequently exceeding 95% accuracy. These datasets primarily feature single, simple intents, which fail to capture the nuance of natural human-car interaction, where users often issue compound commands like "Navigate home and turn on the AC."
  2. Lack of Unified Benchmarking: While researchers are increasingly experimenting with LLMs and LALMs for SLU, the field lacks standardized prompts, formats, and evaluation methodologies. This fragmentation makes direct comparisons between different modeling paradigms difficult.

Introducing the MAC-SLU Dataset

To address these gaps, the researchers developed MAC-SLU, a comprehensive Chinese multi-intent dataset specifically tailored for automotive environments. The dataset comprises over 20,000 commands spanning 8 distinct domains, including Car Control, Map, and Music.

Comparison of MAC-SLU with existing SLU benchmarks, highlighting its superior scale in intents, slots, and multi-intent support.

As shown in the comparison above, MAC-SLU significantly exceeds existing benchmarks like SLURP in complexity, featuring 81 intents and 192 slots. Crucially, it is built for multi-intent queries, containing sentences with up to five independent intents. The audio component was generated using the CosyVoice-2 TTS engine, utilizing diverse speaker embeddings to ensure acoustic variety.

Methodology: Evaluating SLU Paradigms

The study evaluates three distinct architectural approaches to SLU:

  • NLU (Text-only): Transcribed text is fed into LLMs (e.g., Qwen3) for semantic parsing.
  • Pipeline (ASR + NLU): An Automatic Speech Recognition (ASR) model like Whisper transcribes the audio before passing it to an LLM.
  • End-to-End (E2E) SLU: LALMs (e.g., Qwen2.5-Omni, GPT-4o-Audio) process raw audio directly to generate semantic structures, potentially avoiding the "telephone game" effect where ASR errors propagate through the system.

These models were tested using both In-Context Learning (ICL)—providing examples within the prompt—and Supervised Fine-Tuning (SFT) via LoRA (Low-Rank Adaptation).

Results and Analysis

The experimental results highlight a significant performance gap between general-purpose prompting and domain-specific training. While LALMs show surprising potential in zero-shot slot filling when prompts are carefully engineered, their overall accuracy (correctly identifying both intent and all slots) remains below 15% without fine-tuning.

Performance metrics for fine-tuned models on the MAC-SLU benchmark, demonstrating the high accuracy achievable through SFT.

As the data indicates, Supervised Fine-Tuning is essential for high-precision applications. For instance, the Qwen2.5-Omni-7B model achieved an impressive 91.24% intent accuracy after fine-tuning. Furthermore, the E2E approach proved its viability by outperforming similarly sized pipeline systems, particularly in scenarios where ASR error rates were high. In the zero-shot arena, closed-source models like GPT-4o-Audio and Gemini-2.5-Flash continue to lead open-source alternatives in overall accuracy.

Conclusion and Future Directions

MAC-SLU establishes a new frontier for automotive voice interaction, revealing that while modern generative models are powerful, they require specialized tuning to meet the rigorous demands of cabin systems. The authors note that current evaluation metrics based on exact string matching may be too rigid for generative models, which often produce semantically correct but lexically different answers. Future work will focus on developing semantically-aligned evaluation metrics and improving the in-context learning capabilities of LALMs to reduce the reliance on expensive fine-tuning.

Summary updated

Abstract

Spoken Language Understanding (SLU), which aims to extract user semantics to execute downstream tasks, is a crucial component of task-oriented dialog systems. Existing SLU datasets generally lack sufficient diversity and complexity, and there is an absence of a unified benchmark for the latest Large Language Models (LLMs) and Large Audio Language Models (LALMs). This work introduces MAC-SLU, a novel Multi-Intent Automotive Cabin Spoken Language Understanding Dataset, which increases the difficulty of the SLU task by incorporating authentic and complex multi-intent data. Based on MAC-SLU, we conducted a comprehensive benchmark of leading open-source LLMs and LALMs, covering methods like in-context learning, supervised fine-tuning (SFT), and end-to-end (E2E) and pipeline paradigms. Our experiments show that while LLMs and LALMs have the potential to complete SLU tasks through in-context learning, their performance still lags significantly behind SFT. Meanwhile, E2E LALMs demonstrate performance comparable to pipeline approaches and effectively avoid error propagation from speech recognition. Code\footnote{https://github.com/Gatsby-web/MAC\_SLU} and datasets\footnote{huggingface.co/datasets/Gatsby1984/MAC\_SLU} are released publicly.
BibTeX
@misc{peng2025mac,
  author = {Yuezhang Peng and Chonghao Cai and Ziang Liu and Shuai Fan and Sheng Jiang and Hua Xu and Yuxin Liu and Qiguang Chen and Kele Xu and Yao Li and Sheng Wang and Libo Qin and Xie Chen},
  title = {{MAC-SLU: Multi-Intent Automotive Cabin Spoken Language Understanding Benchmark}},
  year = {2025},
  eprint = {2512.01603},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2512.01603},
}
Figure