Cross-Modal Contrastive Representation Learning for Audio-to-Image Generation

Hae-Chun Chung, JooYong Shim, Jong-Kook KimarXiv 2022 · arXiv · OpenReview

TL;DR

This paper introduces Cross-Modal Contrastive Representation Learning (CMCRL) to improve audio-to-image generation. By training audio and image encoders to produce similar latent representations, CMCRL extracts richer audio features that lead to higher-quality generated images compared to previous methods, even without explicit labels.

What Problem Does This Paper Address?

Generating data in one modality from another (cross-modal generation) is challenging because different modalities often have distinct properties, making it difficult to translate information effectively. Specifically, this paper tackles the problem of generating realistic images from audio inputs. While audio and image data are often related (e.g., in videos), directly using audio features to guide image generation can result in lower quality outputs. The goal is to bridge this modality gap to enable more accurate and visually appealing image synthesis from audio.

Methods

The core innovation is the Cross-Modal Contrastive Representation Learning (CMCRL) framework. Unlike traditional classification-based approaches that learn audio features in isolation, CMCRL trains both audio and image encoders simultaneously. It uses a supervised contrastive learning objective to pull the latent representations of corresponding audio and image pairs closer together in an embedding space, while pushing apart representations from different pairs.

Here's a breakdown:

  • Audio and Image Encoders: Pre-trained encoders (like ResNet-18) are used for both modalities.
  • Projection Networks: After the encoders, projection networks map the features to a lower-dimensional space (128-D) where the contrastive loss is applied.
  • Contrastive Loss: This loss function encourages similarity between audio and image features from the same source (positive pairs) and dissimilarity from different sources (negative pairs). This effectively makes the audio features "understand" the visual concepts they should represent.
  • Generative Adversarial Network (GAN): The learned audio features from CMCRL are then used as a condition for a Self-Attention Generative Adversarial Network (SAGAN) to generate the final image. This SAGAN architecture is enhanced with spectral normalization and a two-time-scale update rule for stable training.

The key novelty lies in using contrastive learning across modalities to create more informative audio embeddings, which are then fed into a powerful GAN for generation.

Datasets

The paper uses the Sub-URMP dataset for both training and evaluation. This dataset contains audio files and corresponding images for 13 different musical instruments. The audio clips are 0.5 seconds long.

Key Insights and Ideas

  • Bridging Modality Gaps: Contrastive learning is a powerful tool for aligning representations across different modalities, effectively reducing the "gap" between audio and image feature spaces.
  • Richer Audio Features: By learning audio features in relation to images, CMCRL extracts more semantically rich and visually relevant information than methods relying solely on audio classification.
  • Label-Free Generation: A significant advantage is that the proposed method can generate high-quality images without requiring explicit class labels during the generation phase, relying solely on the audio input.
  • Improved GAN Conditioning: Using CMCRL-derived audio features as conditioning for GANs leads to demonstrably better image quality and accuracy compared to using features from simple classification learning.

Key Results

  • Classification Accuracy: CMCRL significantly outperforms classification learning in extracting features for both audio (93.49% vs. 90.22%) and image (99.99% vs. 95.93%) classification tasks on the Sub-URMP dataset. This validates the quality of the learned representations.
  • Image Generation Quality:
    • FID (Fréchet Inception Distance): The proposed CMCRL + SAGAN method achieves the lowest FID score (107.26), indicating higher quality and more realistic generated images compared to other methods like CMCGAN (215.92) and CAR-GAN (207.37).
  • IS (Inception Score): CMCRL + SAGAN also achieves the highest IS score (5.288), suggesting better diversity and recognizability of generated images.
  • Generated Image Classification Accuracy: The generated images from CMCRL + SAGAN achieve a test accuracy of 0.8907, which is comparable to CAR-GAN (0.9068) but achieved without explicit labels.

Limitations and Future Work

  • Dataset Size: The paper notes that the Sub-URMP dataset has a limited number of images available for qualitative comparison due to the lack of usable code from previous works. This limits the scope of direct visual comparison.
  • Qualitative Evaluation: While quantitative metrics are strong, the qualitative evaluation is somewhat constrained by the availability of generated images from other methods. The authors present generated images for 13 classes and compare them to ground truth and CAR-GAN, showing visually similar and high-quality results.
  • Future Work: The paper implicitly suggests that further exploration of contrastive learning strategies for cross-modal tasks and integration with more advanced GAN architectures could yield even better results. The success in audio-to-image generation also opens doors for applying CMCRL to other cross-modal generation problems.

Summary updated

Abstract

Multiple modalities for certain information provide a variety of perspectives on that information, which can improve the understanding of the information. Thus, it may be crucial to generate data of different modality from the existing data to enhance the understanding. In this paper, we investigate the cross-modal audio-to-image generation problem and propose Cross-Modal Contrastive Representation Learning (CMCRL) to extract useful features from audios and use it in the generation phase. Experimental results show that CMCRL enhances quality of images generated than previous research.
BibTeX
@misc{chung2022cross,
  author = {Hae-Chun Chung and JooYong Shim and Jong-Kook Kim},
  title = {{Cross-Modal Contrastive Representation Learning for Audio-to-Image Generation}},
  year = {2022},
  eprint = {2207.12121},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2207.12121},
}
Figure