Introduction
Modern voice assistants rely on Spoken Language Understanding (SLU) to interpret user intent, emotional state, and the functional role of utterances within a conversation. Traditionally, these tasks—such as intent classification, emotion recognition, and dialog act detection—are handled by separate, specialized models. However, this modular approach suffers from two primary drawbacks: high computational latency and "context blindness," where each utterance is processed in isolation without regard for the preceding conversation.
This paper presents a novel end-to-end (E2E) architecture designed to jointly predict multiple SLU tasks within a single model. By integrating dialogue history and employing an order-agnostic training strategy, the proposed system achieves performance parity with or exceeds specialized models while significantly reducing memory overhead and response time.
Methodology
The core of the proposed system is a Joint E2E SLU model that leverages both current acoustic signals and historical context.
Integrated Dialog History
To address context blindness, the model incorporates a Semantic Encoder based on BERT. This encoder processes the transcripts of previous utterances along with the SLU tags predicted in those turns. By treating previous tags (e.g., "Information Request" or "Greeting") as context, the model gains a structural understanding of the conversation's flow, which is often as informative as the literal words spoken.
Joint Encoder-Decoder Architecture
The architecture utilizes a Conformer-based acoustic encoder to process the current audio input. The resulting acoustic embeddings are concatenated with the history embeddings from the Semantic Encoder. This combined representation is then passed to an autoregressive decoder. To further refine the model, an auxiliary Connectionist Temporal Classification (CTC) loss is applied at intermediate layers of the encoder to improve alignment between the audio and text representations.
Order-Agnostic Training
A significant challenge in multi-task decoding is determining the sequence in which tags should be predicted. Forcing a fixed order (e.g., always predicting "Intent" before "Emotion") can be sub-optimal. The authors introduce an "order-agnostic" training strategy using CTC loss to calculate the mathematically optimal prediction sequence. This allows the model to dynamically prioritize "easier" or more certain tags to provide better context for more ambiguous ones.
Experimental Setup
The model was evaluated using the HarperValleyBank dataset, which contains approximately 23 hours of audio from 1,446 simulated multi-turn dialogs between bank employees and customers. The dataset is richly annotated with four labels per utterance: dialog acts, intents, speaker roles, and emotional valence. This provides a rigorous environment for testing the model's ability to navigate professional, multi-turn interactions.
Results and Discussion
The experimental results demonstrate that joint modeling with context provides substantial benefits across accuracy, efficiency, and latency.
Performance Gains
The inclusion of dialogue history proved transformative for intent classification, where the F1 score jumped from 47.1% (in a context-less baseline) to 86.5%. Dialog Act recognition also saw an improvement, reaching a 58.8% F1 score compared to 55.5% for separate models. Interestingly, the model's "order-agnostic" approach revealed that for initial greetings, the system preferred predicting speaker roles and dialog acts before intent, confirming that the model uses structural cues to resolve ambiguity.
Efficiency and Latency
By consolidating multiple tasks into a single architecture, the researchers reduced the total parameter count by approximately 66%, from 457.6M to 152.7M. This reduction in complexity translated directly to improved performance in production environments: total latency dropped from 4515ms to 1140ms. This nearly 4x speedup is critical for maintaining the natural cadence of human-machine interaction.
Robustness
The system demonstrated high robustness to Automatic Speech Recognition (ASR) errors. Performance using "Oracle" (perfect) transcripts was only marginally better than performance using noisy, automated transcripts, suggesting that the Semantic Encoder effectively filters out ASR jitter to extract meaningful context.
Conclusion
The joint E2E SLU model represents a significant step toward more efficient and context-aware voice assistants. By integrating dialogue history and optimizing the multi-task prediction process, the authors demonstrate that it is possible to achieve superior accuracy with a fraction of the computational cost. Future research may explore distilling knowledge from larger text-only Large Language Models (LLMs) into this speech-centric framework or extending the architecture to more diverse and casual conversational domains.
Summary updated