CLASP: Cross-modal Alignment Using Pre-trained Unimodal Models
Introduction
Cross-modal Alignment Using Pre-trained Unimodal Models (CLASP) is a novel framework designed to align speech and text representations without the need for massive parallel datasets for pre-training. In the field of multimodal AI, "parallel data"—perfectly matched audio and transcripts—is the gold standard but remains a major bottleneck. Collecting and annotating thousands of hours of parallel speech-text data is expensive and often unfeasible for low-resource languages. CLASP addresses this challenge by aligning powerful, existing unimodal models (e.g., RoBERTa for text and wav2vec2 for speech) using only a small amount of task-specific data.
Methodology
The CLASP framework operates through three primary stages:
- Disentanglement via Multi-Level Contrastive Learning: Rather than treating a speech signal as a single vector, CLASP separates it into modality-invariant (semantic meaning) and modality-specific (prosody, tone, accent) components. Contrastive losses at both levels ensure semantic alignment across modalities while preserving distinct modality-specific features.
- Swap Reconstruction Mechanism: This mechanism serves as a validation for successful disentanglement. It attempts to reconstruct the original speech representation by combining the semantic meaning from text with the stylistic features from speech. This process ensures that no critical information is lost during the disentanglement phase.
- Multi-Head Fusion: The model employs a Transformer-based fusion mechanism that treats different attention heads from pre-trained models as distinct features, allowing the model to attend to various patterns (such as emotional cues versus linguistic structures) during final classification.
Datasets
CLASP was evaluated on three major benchmarks:
- IEMOCAP: A standard dataset for emotion recognition in conversation.
- SLURP: A large-scale dataset for spoken intent classification.
- MIST (Multimodal Idiom uSage deTection): A new dataset introduced by the authors, containing 6,325 text-speech pairs from LibriSpeech. It is specifically annotated to distinguish between literal and figurative idiom usage, a task where vocal tone often provides critical context for disambiguation.
Key Results and Insights
CLASP demonstrates superior performance across multiple benchmarks. On the IEMOCAP emotion task, it improved the F1 score by over 6 points compared to text-only models and 10 points compared to speech-only models. It also outperformed existing multimodal fusion models like LFMIM and LRMP by approximately 3 points on the IEMOCAP4 dataset.
On the newly introduced MIST dataset, CLASP achieved an F1 score of 0.83, significantly outperforming unimodal baselines (0.77 for text, 0.73 for audio). This result underscores the importance of "how" something is said in understanding idiomatic meaning.
Robustness and Disentanglement
A key insight from the study is the power of disentanglement. t-SNE visualizations confirm that CLASP successfully separates "what is said" from "how it is said," allowing the model to remain flexible under noisy conditions. CLASP showed remarkable resilience to high Word Error Rates (WER) in text and low Signal-to-Noise Ratios (SNR) in audio, consistently outperforming baselines. Furthermore, the model performs well in modality-scarce scenarios by "hallucinating" missing modalities using TTS or ASR.
Conclusion
CLASP offers a robust and efficient way to align speech and text without massive parallel pre-training. While the model's size—due to stacking two large pre-trained models—presents a computational challenge, its performance and robustness make it a promising approach for real-world applications. Future work will explore parameter-efficient fine-tuning methods like LoRA and further refinement of the MIST dataset through human verification.
Summary updated