Abstract
Word-level alignment between speech and text is a critical component for high-performance multimodal pre-training. However, obtaining ground-truth alignments is labor-intensive and expensive. This paper introduces USDP (Unsupervised Speech-text word-level alignment with Dynamic Programming), a novel framework that eliminates the need for manual annotations. By employing an iterative Expectation-Maximization (EM) inspired approach combined with Dynamic Programming, USDP learns to map audio segments to specific words directly from raw speech-text pairs. Experimental results across multiple downstream tasks, including sentiment analysis and emotion recognition, demonstrate that USDP achieves state-of-the-art performance, matching or exceeding models trained with supervised alignment data.
Introduction
In multimodal dialogue systems, "alignment"—the precise mapping of audio segments to their corresponding written words—is essential for capturing the nuances of human communication. While supervised models like SPECTRA have demonstrated the benefits of word-level alignment, they rely on expensive ground-truth annotations. Existing unsupervised methods often operate at the phoneme level, which can strip away semantic context necessary for complex tasks like intent detection or sentiment analysis. USDP addresses this by learning word-level alignments in an unsupervised manner, bridging the gap between raw data availability and high-level semantic understanding.
Methodology
The USDP framework utilizes an iterative training process inspired by the Expectation-Maximization (EM) algorithm. The architecture consists of three primary phases:
- Estimation: A temporal prediction head makes an initial estimate of word durations within a sentence.
- Alignment: Speech and text are projected into a shared feature space. An association matrix is constructed, and Dynamic Programming (DP) identifies the optimal monotonic path that minimizes reconstruction loss.
- Optimization: The path generated by DP serves as a "pseudo-label" to refine the model's parameters.
A significant innovation is the Dialogue Consistency Loss ($L_{CON}$), which ensures that word-alignment predictions remain stable across different dialogue contexts. The model uses RoBERTa for text encoding and a WavLM-based structure for speech processing.
Experimental Setup
The model was pre-trained on a 960-hour subset of the Spotify 100K English podcast dataset, featuring real-world multi-turn dialogues. Evaluation was conducted across several benchmarks:
- Multimodal Sentiment Analysis (MSA): MOSI and MOSEI datasets.
- Emotion Recognition in Conversation (ERC): MELD and IEMOCAP datasets.
- Spoken Language Understanding (SLU): MIntRec (intent recognition).
- Dialogue State Tracking (DST): SpokenWoZ.
Results and Discussion
USDP consistently outperformed previous state-of-the-art (SOTA) models across all benchmark datasets. Notably, on the MOSI dataset, USDP-large achieved a 7.03 percentage point improvement in 7-class accuracy over the previous SOTA, MIB. In intent recognition (MIntRec), it improved results by over 4 percentage points.
Crucially, USDP achieved parity with SPECTRA, a model trained on supervised word-level annotations. This suggests that unsupervised alignment via Dynamic Programming is a highly effective and scalable alternative to manual labeling. The model's focus on word-level semantics rather than phonemes allows it to capture the rich information needed for high-level reasoning, such as sarcasm or emotional shifts.
Conclusion
USDP demonstrates that unsupervised word-level alignment is not only feasible but highly competitive with supervised methods. By leveraging Dynamic Programming as a corrective teacher, the model learns robust cross-modal mappings from raw data. Future work will focus on scaling these alignments to a "universal" level similar to image-text models like CLIP, exploring longer dialogue histories, and integrating Large Language Models (LLMs) to further enhance contextual learning.
Summary updated