Abstract
Spoken Language Understanding (SLU) consists of two sub-tasks: intent detection (ID) and slot filling (SF). Given its broad range of real-world applications, enhancing SLU for practical deployment is increasingly critical. Profile-based SLU addresses ambiguous user utterances by incorporating context awareness (CA), user profiles (UP), and knowledge graphs (KG) to support disambiguation, thereby advancing SLU research toward real-world applicability. However, existing SLU datasets still fall short in representing real-world scenarios. Specifically, (1) CA uses one-hot vectors for representation, which is overly idealized, and (2) models typically focuses solely on predicting intents and slot labels, neglecting the reasoning process that could enhance performance and interpretability. To overcome these limitations, we introduce VRSLU, a novel SLU dataset that integrates both Visual images and explicit Reasoning. For over-idealized CA, we use GPT-4o and FLUX.1-dev to generate images reflecting users' environments and statuses, followed by human verification to ensure quality. For reasoning, GPT-4o is employed to generate explanations for predicted labels, which are then refined by human annotators to ensure accuracy and coherence. Additionally, we propose an instructional template, LR-Instruct, which first predicts labels and then generates corresponding reasoning. This two-step approach helps mitigate the influence of reasoning bias on label prediction. Experimental results confirm the effectiveness of incorporating visual information and highlight the promise of explicit reasoning in advancing SLU.
1. Introduction
Spoken Language Understanding (SLU) is the core technology behind modern virtual assistants. Traditionally, this involves Intent Detection (ID) and Slot Filling (SF). However, real-world requests are frequently ambiguous. For example, a request like "Play Martial Universe" could refer to a song, a television series, or an audiobook depending on the context.
While "Profile-based SLU" has attempted to address this by providing user context, it suffers from two primary limitations: the use of over-idealized "one-hot" vectors to represent complex environments and the lack of interpretability in model outputs. To bridge this gap, we present VRSLU, a benchmark that moves beyond text-only context to include realistic visual scenes and explicit reasoning.
2. The VRSLU Dataset
The VRSLU (Visual scenes and Reasoning SLU) dataset is the first of its kind to combine multimodal context with logical rationales. It builds upon the ProSLU dataset, adding over 10,000 human-verified images and expert-refined reasoning chains.
3. Methodology
Our methodology focuses on creating a high-fidelity multimodal environment and a robust prompting strategy to ensure both accuracy and interpretability.
3.1 Visual Scene Construction
To move beyond simple categorical tags, we utilize a generative pipeline to create realistic visual context. GPT-4o transforms raw context data into descriptive paragraphs, which are then processed by the FLUX.1-dev image generation model. Crucially, these images undergo manual verification to ensure they accurately represent the user's environment (e.g., being at home vs. at work) and status (e.g., walking vs. lying down).
3.2 Reasoning and the LR-Instruct Template
We generate logical explanations for every intent and slot label using GPT-4o, followed by manual refinement. To address the risk of "reasoning bias"—where flawed logic can lead a model to an incorrect final answer—we introduce LR-Instruct (Labels-then-Reasoning). This multi-turn prompt requires the model to predict labels first and provide reasoning second, preventing logical hallucinations from corrupting the primary classification task.
4. Key Results and Insights
Experimental evaluations using Multimodal Large Language Models (MLLMs) such as Qwen2-VL and Yi-VL demonstrate the following:
- Visual Context is Critical: Ablation studies show that removing visual images leads to a significant drop in accuracy (approx. 3.39%), proving that images provide a richer, more effective context than simple vectors.
- Performance Gains: The Qwen2.5-VL-7B model, when paired with the LR-Instruct method, achieved a state-of-the-art accuracy of 86.63%.
- Enhanced Interpretability: By requiring explicit reasoning, models are forced to attend to relevant details in user profiles and knowledge graphs, making their decision-making process transparent and easier to debug.
5. Conclusion
VRSLU provides a more realistic and challenging benchmark for the SLU community. By integrating visual scenes and structured reasoning, we demonstrate that multimodal context and "labels-first" prompting are essential for building reliable, interpretable virtual assistants for real-world deployment.
Summary updated