Abstract
Deep neural networks are usually data-starved, but manual annotation can be costly in many specific tasks, such as emotion recognition from audio. However, there is a large amount of publicly available labeled image-based facial expression recognition datasets. How these images could help audio emotion recognition with limited labeled data based on their inherent correlations is a meaningful and challenging task. In this paper, we propose a semi-supervised adversarial network that allows knowledge transfer from labeled videos to the heterogeneous labeled audio domain, enhancing audio emotion recognition performance. Specifically, face image samples are translated to spectrograms class-wisely. To harness the translated samples in a sparsely distributed area and construct a tighter decision boundary, we propose to precisely estimate the density on feature space and incorporate reliable low-density samples with an annealing scheme. Moreover, unlabeled audios are collected with a high-density path in a graph representation. As a possible "recognition via generation" framework, we empirically demonstrated its effectiveness on several audio emotional recognition benchmarks and its generality on recent large-scale semi-supervised domain adaptation tasks.
Introduction
Deep learning models are notoriously data-hungry. In emotion recognition, labeling audio data (Audio Emotion Recognition, or AER) is particularly expensive and time-consuming, resulting in small, sparse datasets. Conversely, Image-based Facial Expression Recognition (IFER) datasets are relatively abundant.
The primary challenge lies in the fact that these two domains—facial images and vocal spectrograms—are heterogeneous, possessing completely different data structures. While standard domain adaptation typically bridges similar domains (e.g., photos to sketches), this work addresses Semi-supervised Domain Adaptation (SSDA) across different modalities, leveraging labeled images to improve a classifier that ultimately operates on audio.
Proposed Methodology
The proposed "recognition via generation" framework utilizes a multi-stage architecture to bridge the gap between image and audio domains:
- Cross-Modal Translation: A VAE-GAN (Variational Autoencoder + Generative Adversarial Network) backbone translates face images into audio spectrograms. This translation is "classification-aware," ensuring that the emotional content (e.g., a "happy" face) is preserved in the generated spectrogram.
- Density-Based Sample Handling: To refine decision boundaries, the model distinguishes between high-density samples (typical class examples used for data augmentation) and low-density samples (edge cases near boundaries). The latter are used in an adversarial game to force the classifier to find a more robust boundary.
- Reliability Path: Unlabeled audio data is incorporated via a graph representation where labels are propagated along "high-density paths," safely assigning labels to unlabeled samples based on similarity.
- Annealing Scheme: A hyperparameter ($\epsilon$) gradually increases during training, incorporating more generated samples as the model's confidence grows.
Experimental Results
The framework was evaluated on several key benchmarks, including audio-visual datasets like CREMA-D and RAVDESS, and facial expression datasets such as CK+ and MMI.
Key Findings
- Significant Accuracy Gains: On the CREMA-D dataset, the baseline accuracy of 30.81% (using only labeled audio) jumped to 58.71% when incorporating IFER data through the proposed framework.
- Robustness in SSDA: On the DomainNet benchmark, the method consistently outperformed state-of-the-art SSDA methods like Minimax Entropy (MME), particularly when labeled target samples were scarce.
- Feature Discriminability: Quantitative analysis confirmed that the learned features were more compact and discriminative than standard baselines.
- Unpaired Learning: Unlike previous methods requiring matched audio-visual pairs, this framework can leverage any facial dataset to assist any audio dataset, regardless of whether they feature the same subjects.
Conclusion
This work demonstrates that "bad" generators—those producing samples in low-density gaps between classes—can actually create better classifiers by forcing more precise decision boundaries. While density estimation and graph construction add computational complexity, the ability to leverage heterogeneous data sources offers a powerful solution for data-starved modalities. Future work may extend this "recognition via generation" approach to other cross-modal tasks, such as text-to-image or multi-sensor robotics data.
Summary updated