TL;DR
This paper introduces "Turbo," a novel contrastive learning strategy that enhances multi-modal classification by jointly optimizing in-modal and cross-modal representations. By leveraging a simple yet effective technique of passing data through the model twice with different dropout masks, Turbo generates multiple contrastive objectives, leading to significant performance improvements on audio-text classification tasks.
What Problem Does This Paper Address?
Multi-modal representation learning, particularly for tasks involving audio and text, has seen significant advancements with contrastive learning. However, existing methods primarily focus on learning alignments between different modalities (cross-modal understanding), often neglecting the importance of learning robust representations within each modality (in-modal understanding). This oversight can limit the overall performance, as a strong understanding of individual modalities is crucial for effective fusion and classification. Furthermore, many state-of-the-art approaches rely on extensive pre-training on massive paired datasets, which can be difficult and costly to acquire, especially in specialized domains. This paper aims to bridge this gap by proposing a method that simultaneously optimizes both in-modal and cross-modal representations and integrates this self-supervised learning directly into the supervised classification task, reducing the reliance on massive pre-training datasets.
Methods
The core innovation of this paper is the "Turbo" method, a self-supervised contrastive learning strategy. The key idea is to pass each multi-modal data pair (e.g., audio and text) through the model's encoders twice within a single training step. Crucially, different "hidden dropout masks" are applied during these two forward passes. This randomness from dropout, typically used for regularization, is repurposed here as a form of data augmentation to generate distinct representations for each modality from the same input.
With these two sets of representations for each modality (e.g., two audio representations, h_a1 and h_a2), Turbo formulates several contrastive learning objectives:
- In-modal Contrastive Learning: This objective pulls the two representations of the same modality (e.g.,
h_a1andh_a2) closer together, while pushing them away from other representations of the same modality in the batch. This enhances the learning of robust uni-modal features. - Cross-modal Contrastive Learning: This objective learns the alignment between different modalities. Since Turbo generates two representations for each modality, it creates multiple cross-modal pairs (e.g.,
h_a1withh_t1,h_a1withh_t2,h_a2withh_t1,h_a2withh_t2). These pairs are used to train contrastive objectives that pull corresponding audio and text representations together and push non-corresponding ones apart.
The final Turbo loss is a sum of these in-modal and cross-modal losses. This self-supervised Turbo loss is then combined with the standard supervised classification loss (e.g., cross-entropy) using a balancing hyperparameter, allowing for joint training.
Datasets
The paper evaluates the proposed Turbo method on two audio-text classification tasks:
- Speech Emotion Recognition (SER): The IEMOCAP dataset is used. This is a well-established multi-modal emotion recognition benchmark dataset containing audio and video recordings of dyadic interactions. The paper uses the audio and text modalities from IEMOCAP.
- Device-directed Speech Detection (DSD): For this task, the authors use a real-life, in-house dataset named REJ. This dataset consists of audio utterances annotated as either device-directed (e.g., voice commands for a virtual assistant) or non-device-directed. The paper notes that no publicly available data for this specific task was used.
Key Insights and Ideas
- Dropout as Augmentation for Contrastive Learning: Repurposing dropout, a standard regularization technique, as a mechanism to generate diverse representations for contrastive learning is a clever and efficient approach. It avoids the need for complex, separate data augmentation pipelines.
- Joint In-modal and Cross-modal Optimization: The paper convincingly argues that neglecting in-modal contrastive learning is a missed opportunity. By jointly optimizing both, Turbo achieves a more comprehensive understanding of multi-modal data.
- Multiple Contrastive Objectives from a Single Pass: The "forward twice" strategy elegantly generates multiple in-modal and cross-modal contrastive objectives from a single input pair, maximizing the learning signal without significantly increasing computational cost per forward pass.
- Direct Integration with Supervised Learning: Combining the self-supervised Turbo loss directly with the supervised classification loss, rather than relying solely on pre-training, makes the method more practical and adaptable to domains with limited paired data.
Key Results
The Turbo method demonstrates significant improvements across both evaluated tasks:
- Speech Emotion Recognition (IEMOCAP): Turbo achieved a weighted accuracy (WA) of 0.7709 and unweighted accuracy (UA) of 0.7635. This represents an improvement of 5.59% in WA and 5.32% in UA compared to a vanilla baseline (direct concatenation of features) and 5.59% and 5.32% over a baseline with only cross-modal contrastive learning. The paper also reports state-of-the-art performance on the IEMOCAP benchmark.
- Device-directed Speech Detection (REJ): Turbo improved accuracy by 3.83% and reduced the Equal Error Rate (EER) by 2.62% compared to the vanilla baseline.
- Qualitative Analysis: Visualizations of learned representations (alignment and uniformity plots) show that Turbo effectively improves both the alignment between modalities and the uniformity of representations within each modality, which correlates with performance gains.
Limitations and Future Work
- Limitations: The paper primarily focuses on audio-text modalities. While the method is presented as generalizable, its effectiveness on other modality combinations (e.g., vision-language) is not explored in this work. The balancing hyperparameter
λfor combining losses is set empirically. - Future Work: The authors suggest extending Turbo to audio-text pre-training for downstream tasks. They also express interest in expanding the method to include the visual domain, aiming for a three-modality classification system.
Summary updated