TL;DR
This paper introduces a novel "Cross-modal Semantic Alignment before Fusion" (CSAF) method for end-to-end spoken language understanding (SLU). By aligning speech and text representations in a shared semantic space before fusing them, CSAF significantly improves intent classification accuracy, especially in challenging scenarios with noisy transcriptions.
What Problem Does This Paper Address?
Spoken Language Understanding (SLU) aims to extract semantic meaning (like user intent) directly from spoken audio. Traditional SLU systems often use a cascaded approach: first transcribing speech to text (ASR), then understanding the text (NLU). This is prone to errors from the ASR stage and loses valuable acoustic information. End-to-end (E2E) SLU models aim to overcome these limitations by processing audio directly.
However, existing E2E SLU models, particularly two-pass architectures that refine initial predictions, often simply fuse speech and text embeddings without explicitly aligning their underlying semantic representations. This can lead to suboptimal performance, especially when the initial ASR transcriptions are imperfect. The paper addresses the challenge of effectively merging information from speech and text modalities in a way that leverages their semantic similarities before the fusion step.
Methods
The core innovation is the Cross-modal Semantic Alignment before Fusion (CSAF) approach. It's integrated into a two-pass E2E SLU architecture.
- Shared Semantic Memory Transformer: To align speech and text embeddings into a common semantic space, the authors propose a novel transformer module. This module uses "memory queries" to project uni-modal embeddings (from speech and text) into a fixed-length representation. Crucially, this alignment happens before the main fusion.
- Contrastive Loss (Lcon): A contrastive loss is applied to the aligned semantic memories. This loss encourages representations of semantically similar speech and text segments to be closer in the shared space, while pushing dissimilar ones apart. This is a sequence-level alignment, aiming to capture richer semantic relationships than token-level alignment.
- Gated Multi-modal Network: After alignment, a gated network dynamically controls the contribution of each modality's aligned representation to the final fused embedding. This allows the model to weigh the importance of speech versus text information based on the context.
- Two-Pass Architecture: The overall model follows a two-pass structure. The first pass generates an initial intent and transcription. The second pass refines these predictions using the fused, aligned representations. The speech encoder is frozen in the second pass to maintain consistency.
Datasets
The proposed CSAF method was evaluated on two publicly available datasets:
- FSC Challenge: A benchmark dataset for SLU evaluation, consisting of spoken commands for smart home control. The authors specifically used the Challenge Utterance Set and Challenge Speaker Set, which feature unseen speakers and utterances, respectively, to test generalization.
- SLURP: A more complex dataset with richer lexical and syntactic structures. The authors also used an additional synthesized version (SLURP-Synth) for evaluation.
Key Insights and Ideas
- Alignment Before Fusion is Crucial: The paper strongly suggests that explicitly aligning speech and text representations into a shared semantic space before fusing them is more effective than simply concatenating or attending to them independently.
- Sequence-Level Contrastive Loss: Using a sequence-level contrastive loss on semantic memories, rather than token-level alignment, proves more effective for capturing utterance-level semantic relationships.
- Robustness to ASR Errors: The CSAF method demonstrates improved robustness to ASR errors, which is a significant advantage for real-world SLU systems where transcription accuracy can vary. This is highlighted by its performance on the FSC dataset, where ASR transcripts have higher error rates.
- Shared Semantic Memory Transformer: This novel component effectively bridges the modality gap by projecting diverse embeddings into a common, fixed-length semantic space using a limited number of shared layers.
Key Results
- Significant Accuracy Gains: CSAF achieved an absolute improvement of 3.1% in intent classification accuracy on the FSC Challenge Utterance Set compared to previous two-pass models. It also showed improvements on the SLURP dataset.
- Competitive Performance: The model achieved competitive results on the FSC Challenge Speaker Set, indicating good generalization to unseen speakers.
- Qualitative Improvement: The method notably reduced misclassifications of phrases like "open language settings" into incorrect intents, a common issue for previous models.
Limitations and Future Work
- Low-Resource Scenarios: The authors suggest exploring how to leverage pre-training tasks to further improve performance in low-resource scenarios.
- ASR Degradation: While CSAF is more robust to ASR errors, performance can still be impacted by high ASR Word Error Rates (WER), as observed on the FSC dataset. Future work could explore more advanced ASR error mitigation strategies.
- Computational Cost: While not explicitly detailed as a limitation, the addition of a shared semantic memory transformer and contrastive loss likely increases training complexity and computational cost.
The paper presents a promising direction for improving E2E SLU by focusing on semantic alignment as a critical pre-fusion step.
Summary updated