Abstract
Analyzing real-world multimodal signals is an essential and challenging task for intelligent voice assistants (IVAs). Mainstream approaches have achieved remarkable performance on various downstream tasks of IVAs with pre-trained audio models and text models. However, these models are pre-trained independently and usually on tasks different from target domains, resulting in sub-optimal modality representations for downstream tasks. Moreover, in many domains, collecting enough language-audio pairs is extremely hard, and transcribing raw audio also requires high professional skills, making it difficult or even infeasible to joint pre-training. To address these painpoints, we propose DSCLAP, a simple and effective framework that enables language-audio pre-training with only raw audio signal input. Specifically, DSCLAP converts raw audio signals into text via an ASR system and combines a contrastive learning objective and a language-audio matching objective to align the audio and ASR transcriptions. We pre-train DSCLAP on 12,107 hours of in-vehicle domain audio. Empirical results on two downstream tasks show that while conceptually simple, DSCLAP significantly outperforms the baseline models in all metrics, showing great promise for domain-specific IVAs applications.
Introduction
Intelligent Voice Assistants (IVAs) must navigate the complex intersection of acoustics and semantics. While multimodal pre-training has seen success in vision-language domains (e.g., CLIP), the audio-language field remains hindered by data scarcity. Current state-of-the-art systems often rely on generic models like BERT for text and Wav2vec2 for audio, which are fused post-hoc. This approach suffers from two primary drawbacks: semantic disconnection, where modalities are learned independently, and a significant domain gap between generic training data and specialized environments, such as noisy vehicle cabins.
The cost of manual transcription for domain-specific audio is often prohibitive. DSCLAP (Domain-Specific Contrastive Language-Audio Pre-Training) addresses this by enabling high-quality multimodal alignment using only raw, unlabeled audio.
Methodology
The DSCLAP framework utilizes a novel pipeline to bridge the gap between audio and text without human intervention.
- Automated Transcription: Raw audio is processed through an Automatic Speech Recognition (ASR) system to generate "pseudo-transcripts." Despite an 18.7% character error rate, these transcripts provide a sufficient linguistic anchor for the audio signal.
- Dual Encoder Architecture: The system employs Wav2vec2 for acoustic feature extraction and TinyBERT for linguistic features.
- Contrastive Learning (InfoNCE): The model maximizes similarity between audio clips and their corresponding ASR transcripts in a shared latent space.
- Language-Audio Matching (LAM): To enhance alignment efficiency, a LAM objective with hard negative sampling is introduced. This forces the model to distinguish between highly similar but non-matching pairs, refining the semantic boundaries.
Experimental Setup
The model was pre-trained on a massive in-vehicle dataset comprising 12,107 hours of audio (over 20 million clips) from 250,000 vehicles. Evaluation was conducted on two downstream tasks:
- Multimodal Device-directed Speech Detection (MDSD): Distinguishing user-to-car commands from background speech.
- Multimodal Conversational Intent Classification (MCIC): Categorizing user goals into 15 distinct classes (e.g., navigation, media control).
Results and Discussion
DSCLAP demonstrated superior performance across all metrics compared to strong baselines like Whisper and Wav2vec2-BERT combinations.
Performance Gains
On the MDSD task, DSCLAP achieved an accuracy of 95.06%, a significant improvement over the 92.48% baseline. Crucially, the False Rejection Rate (FRR) was halved from 5.35% to 2.77%, directly translating to a more reliable user experience. For intent classification (MCIC), DSCLAP reached 89.98% accuracy, outperforming specialized optimization techniques like RDrop.
Data Efficiency and the "Cold Start" Solution
One of the most striking findings was DSCLAP's performance in low-data regimes. The model exhibits a remarkable "cold start" capability, where pre-training on raw audio allows it to achieve high accuracy with very few downstream labels.
As shown in the figure above, DSCLAP trained on only 1,000 samples outperformed a baseline model trained on 10,000 samples. This efficiency makes it an ideal candidate for deploying IVAs in new domains where labeled data is scarce.
Conclusion
DSCLAP proves that imperfect, ASR-generated transcripts are a powerful resource for multimodal pre-training. By embracing noisy labels and utilizing hard negative sampling, the framework achieves state-of-the-art results in domain-specific tasks while drastically reducing the need for manual transcription. Future work will explore the limits of ASR-based pre-training in highly technical domains and methods to further bridge the gap between noisy pre-training and clean fine-tuning.
Summary updated