Abstract
This paper details the winning approach for the ICASSP 2023 Spoken Task-Oriented Parsing (STOP) Quality Challenge. Researchers from Carnegie Mellon University and Sony Group Corporation achieved first place by integrating end-to-end (E2E) and pipeline Spoken Language Understanding (SLU) systems. By combining fine-tuned Whisper models, large-scale language models like BART, and a robust system combination strategy using ROVER, the team achieved a state-of-the-art 80.8% Exact Match (EM) accuracy. This work highlights the current performance gap between E2E and pipeline architectures and demonstrates the power of ensembling diverse modeling paradigms.
Introduction
Spoken Language Understanding (SLU) is a critical component of modern voice assistants, responsible for extracting structured meaning directly from human speech. The Spoken Task-Oriented Parsing (STOP) task specifically requires systems to convert spoken utterances into complex semantic trees. Traditionally, this has been approached via a pipeline: Automatic Speech Recognition (ASR) followed by Natural Language Understanding (NLU). While pipelines benefit from massive text-based pre-trained models, they are susceptible to error propagation where ASR mistakes derail the NLU component.
End-to-End (E2E) systems attempt to mitigate this by mapping audio directly to semantic parses. However, E2E models often struggle to match the linguistic reasoning capabilities of large-scale text models. This study explores the integration of both paradigms to maximize performance for the STOP Quality Challenge.
Methodology
The research team evaluated several architectural configurations to identify the strengths of different SLU approaches.
End-to-End (E2E) Architectures
- SSL Frontends: The team utilized self-supervised learning (SSL) models, specifically WavLM and Hubert, as feature extractors for a Conformer-Transformer encoder-decoder architecture.
- Fine-tuned Whisper: Moving beyond frozen frontends, the researchers fine-tuned the full Whisper model (medium and large variants). Whisper’s extensive pre-training on diverse audio proved to be a superior foundation for the SLU task.
- Advanced Decoding Strategies:
- 2-Pass (Deliberation) Model: This architecture uses a second decoder pass to refine outputs by attending to both acoustic features and initial ASR hypotheses.
- Compositional SLU: This model predicts intermediate token representations before generating the final parse, enforcing a more structured understanding of the input.
Pipeline Architectures
The pipeline approach paired high-performing ASR models with large-scale pre-trained language models (PLMs). The team utilized BART-large and T5-large for the NLU stage, leveraging their deep linguistic knowledge to parse the text generated by the ASR component.
System Combination
To capitalize on the diverse strengths of these models, the team employed Recognizer Output Voting Error Reduction (ROVER). This strategy combined the hypotheses from the four best-performing models—encompassing both E2E and pipeline architectures—to produce a final, optimized result.
Experimental Setup
The models were trained and evaluated using the STOP Dataset, the primary benchmark for task-oriented semantic parsing. To enhance the ASR component of the pipeline systems, the researchers utilized external datasets, including LibriSpeech and Common Voice, for pre-training. Training optimizations, such as maintaining original text casing and increasing checkpoint frequency, were also implemented to improve Word Error Rate (WER).
Results and Discussion
The integrated system achieved 80.8% Exact Match (EM) accuracy on the STOP challenge test set, securing first place in the competition.
Performance Comparison
- ASR Quality: The best ASR ensemble reached a WER of 2.2%, a significant improvement over the 4.26% benchmark baseline.
- E2E vs. Pipeline: The best individual pipeline model (ASR + BART-Large) achieved 80.5% EM, while the best individual E2E model (Whisper-large) reached 79.1% EM.
These results suggest that while E2E models like Whisper are closing the gap, pipeline systems currently retain an edge due to the sophisticated reasoning capabilities of large text-based LMs. However, the jump from 80.5% to 80.8% EM through ROVER demonstrates that E2E and pipeline models often make different types of errors, making them highly complementary in an ensemble.
Conclusion
The success of this approach in the ICASSP 2023 STOP Quality Challenge underscores the importance of hybrid strategies in SLU. While Whisper has emerged as a powerful tool for E2E understanding, the linguistic depth of pipeline-based NLU remains a formidable benchmark. Future work will focus on integrating Whisper into more complex E2E architectures, such as 2-pass and compositional models, to determine if the acoustic strength of Whisper can eventually surpass the reasoning power of text-only PLMs.
Summary updated