DSCLAP: Domain-Specific Contrastive Language-Audio Pre-Training

Shengqiang Liu, Da Liu, Anna Wang, Zhiyu Zhang, Jie Gao, Yali Li

Abstract

Analyzing real-world multimodal signals is an essential and challenging task for intelligent voice assistants (IVAs). Mainstream approaches have achieved remarkable performance on various downstream tasks of IVAs with pre-trained audio models and text models. However, these models are pre-trained independently and usually on tasks different from target domains, resulting in sub-optimal modality representations for downstream tasks. Moreover, in many domains, collecting enough language-audio pairs is extremely hard, and transcribing raw audio also requires high professional skills, making it difficult or even infeasible to joint pre-training. To address these painpoints, we propose DSCLAP, a simple and effective framework that enables language-audio pre-training with only raw audio signal input. Specifically, DSCLAP converts raw audio signals into text via an ASR system and combines a contrastive learning objective and a language-audio matching objective to align the audio and ASR transcriptions. We pre-train DSCLAP on 12,107 hours of in-vehicle domain audio. Empirical results on two downstream tasks show that while conceptually simple, DSCLAP significantly outperforms the baseline models in all metrics, showing great promise for domain-specific IVAs applications.

Introduction

Intelligent Voice Assistants (IVAs) must navigate the complex intersection of acoustics and semantics. While multimodal pre-training has seen success in vision-language domains (e.g., CLIP), the audio-language field remains hindered by data scarcity. Current state-of-the-art systems often rely on generic models like BERT for text and Wav2vec2 for audio, which are fused post-hoc. This approach suffers from two primary drawbacks: semantic disconnection, where modalities are learned independently, and a significant domain gap between generic training data and specialized environments, such as noisy vehicle cabins.

The cost of manual transcription for domain-specific audio is often prohibitive. DSCLAP (Domain-Specific Contrastive Language-Audio Pre-Training) addresses this by enabling high-quality multimodal alignment using only raw, unlabeled audio.

Methodology

The DSCLAP framework utilizes a novel pipeline to bridge the gap between audio and text without human intervention.

  1. Automated Transcription: Raw audio is processed through an Automatic Speech Recognition (ASR) system to generate "pseudo-transcripts." Despite an 18.7% character error rate, these transcripts provide a sufficient linguistic anchor for the audio signal.
  2. Dual Encoder Architecture: The system employs Wav2vec2 for acoustic feature extraction and TinyBERT for linguistic features.
  3. Contrastive Learning (InfoNCE): The model maximizes similarity between audio clips and their corresponding ASR transcripts in a shared latent space.
  4. Language-Audio Matching (LAM): To enhance alignment efficiency, a LAM objective with hard negative sampling is introduced. This forces the model to distinguish between highly similar but non-matching pairs, refining the semantic boundaries.
The DSCLAP framework architecture, illustrating the flow from raw audio to ASR and the dual-objective training process.

Experimental Setup

The model was pre-trained on a massive in-vehicle dataset comprising 12,107 hours of audio (over 20 million clips) from 250,000 vehicles. Evaluation was conducted on two downstream tasks:

  • Multimodal Device-directed Speech Detection (MDSD): Distinguishing user-to-car commands from background speech.
  • Multimodal Conversational Intent Classification (MCIC): Categorizing user goals into 15 distinct classes (e.g., navigation, media control).

Results and Discussion

DSCLAP demonstrated superior performance across all metrics compared to strong baselines like Whisper and Wav2vec2-BERT combinations.

Performance Gains

On the MDSD task, DSCLAP achieved an accuracy of 95.06%, a significant improvement over the 92.48% baseline. Crucially, the False Rejection Rate (FRR) was halved from 5.35% to 2.77%, directly translating to a more reliable user experience. For intent classification (MCIC), DSCLAP reached 89.98% accuracy, outperforming specialized optimization techniques like RDrop.

Data Efficiency and the "Cold Start" Solution

One of the most striking findings was DSCLAP's performance in low-data regimes. The model exhibits a remarkable "cold start" capability, where pre-training on raw audio allows it to achieve high accuracy with very few downstream labels.

Comparison of accuracy vs. dataset size, showing DSCLAP's superior performance in low-data scenarios compared to the baseline.

As shown in the figure above, DSCLAP trained on only 1,000 samples outperformed a baseline model trained on 10,000 samples. This efficiency makes it an ideal candidate for deploying IVAs in new domains where labeled data is scarce.

Conclusion

DSCLAP proves that imperfect, ASR-generated transcripts are a powerful resource for multimodal pre-training. By embracing noisy labels and utilizing hard negative sampling, the framework achieves state-of-the-art results in domain-specific tasks while drastically reducing the need for manual transcription. Future work will explore the limits of ASR-based pre-training in highly technical domains and methods to further bridge the gap between noisy pre-training and clean fine-tuning.

Summary updated

Abstract

Analyzing real-world multimodal signals is an essential and challenging task for intelligent voice assistants (IVAs). Mainstream approaches have achieved remarkable performance on various downstream tasks of IVAs with pre-trained audio models and text models. However, these models are pre-trained independently and usually on tasks different from target domains, resulting in sub-optimal modality representations for downstream tasks. Moreover, in many domains, collecting enough language-audio pairs is extremely hard, and transcribing raw audio also requires high professional skills, making it difficult or even infeasible to joint pre-training. To address these painpoints, we propose DSCLAP, a simple and effective framework that enables language-audio pre-training with only raw audio signal input. Specifically, DSCLAP converts raw audio signals into text via an ASR system and combines a contrastive learning objective and a language-audio matching objective to align the audio and ASR transcriptions. We pre-train DSCLAP on 12,107 hours of in-vehicle domain audio. Empirical results on two downstream tasks show that while conceptually simple, DSCLAP significantly outperforms the baseline models in all metrics, showing great promise for domain-specific IVAs applications.
BibTeX
@misc{liu2024dsclap,
  author = {Shengqiang Liu and Da Liu and Anna Wang and Zhiyu Zhang and Jie Gao and Yali Li},
  title = {{DSCLAP: Domain-Specific Contrastive Language-Audio Pre-Training}},
  year = {2024},
  eprint = {2409.09289},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2409.09289},
}
Figure