Abstract
Contrastive learning has become one of the most impressive approaches for multi-modal representation learning. However, previous multi-modal works mainly focused on cross-modal understanding, ignoring in-modal contrastive learning, which limits the representation of each modality. In this paper, we propose a novel contrastive learning strategy, called $Turbo$, to promote multi-modal understanding by joint in-modal and cross-modal contrastive learning. Specifically, multi-modal data pairs are sent through the forward pass twice with different hidden dropout masks to get two different representations for each modality. With these representations, we obtain multiple in-modal and cross-modal contrastive objectives for training. Finally, we combine the self-supervised Turbo with the supervised multi-modal classification and demonstrate its effectiveness on two audio-text classification tasks, where the state-of-the-art performance is achieved on a speech emotion recognition benchmark dataset.
Introduction
In the field of multi-modal learning, such as CLIP or CLAP, the standard paradigm involves aligning different modalities—ensuring that an image of a dog and the text "dog" are mapped to proximal locations in a shared latent space. This is known as cross-modal contrastive learning.
However, this approach often overlooks in-modal learning. By focusing solely on inter-modal relationships, models may fail to capture the intrinsic structure within a single modality (e.g., the nuances of audio signals). Furthermore, traditional contrastive methods typically require vast amounts of paired data for pre-training, which is often unavailable in specialized domains. The proposed Turbo strategy aims to integrate the benefits of contrastive learning directly into supervised classification pipelines without requiring additional data.
Methodology
The core of the Turbo strategy lies in a joint optimization of in-modal and cross-modal relationships through a unique training procedure.
The process consists of three main components:
- The Double Forward Pass: Instead of a single pass, each audio-text pair is processed by the encoders twice.
- Dropout as Augmentation: By leveraging the inherent randomness of dropout masks within the neural network, the two passes generate two distinct feature vectors for the same input. This acts as a "free" data augmentation technique.
- Multi-Headed Loss Function: The model is trained using a composite loss:
- In-modal Loss: Encourages similarity between the two versions of the same modality (e.g., Audio Pass 1 vs. Audio Pass 2) while distinguishing them from other samples in the batch.
- Cross-modal Loss: Optimizes four alignment combinations between the two passes of audio and text features.
- Supervised Loss: A standard classification loss (e.g., Cross-Entropy) is applied to the fused representations to perform the primary task.
Experimental Setup
Turbo was evaluated on two distinct audio-text classification tasks:
- IEMOCAP: A public benchmark for Speech Emotion Recognition (SER), containing ~5,500 utterances across categories like happy, sad, angry, and neutral.
- REJ: A large-scale, proprietary dataset (50k utterances) for "Device-directed Speech Detection," identifying whether speech is intended for a voice assistant.
Discussion and Insights
A key insight of this work is that dropout serves as an effective augmenter. Unlike traditional contrastive learning which requires complex manual augmentations (e.g., noise injection or cropping), Turbo utilizes the network's internal stochasticity to create positive pairs.
Furthermore, the inclusion of in-modal contrastive learning addresses representation collapse. By forcing the model to distinguish between different samples within the same modality, the resulting feature space is more evenly distributed, improving uniformity. This structural improvement makes the final supervised classification task significantly easier for the model.
Results
The Turbo method achieved significant performance gains:
- State-of-the-Art Performance: On IEMOCAP, Turbo reached a Weighted Accuracy (WA) of 76.35%, setting a new benchmark.
- Baseline Improvements: Compared to a vanilla multi-modal baseline, Turbo improved accuracy by 5.59% on emotion recognition and 3.83% on device-directed speech detection.
- Enhanced Alignment and Uniformity: Qualitative analysis using KDE plots confirms that Turbo produces tighter alignment between modalities and a more uniform feature distribution than standard cross-modal methods.
Conclusion and Future Work
While Turbo demonstrates strong results, the reliance on an in-house dataset (REJ) for one of the tasks limits external reproducibility for that specific benchmark. Future research will explore expanding Turbo to tri-modal systems (including video) and investigating its potential for large-scale self-supervised pre-training to develop more robust foundation models.
Summary updated