Enhance audio generation controllability through representation similarity regularization

Yangyang Shi, Gaël Le Lan, Varun Nagaraja, Zhaoheng Ni, Xinhao Mei, Ernie Chang, Forrest N. Iandola, Yang Liu, Vikas ChandraarXiv 2023 · DOI · OpenReview

Introduction

Meta AI researchers have introduced Representation Similarity Regularization (RSR), a training technique designed to address the "prompt neglect" problem in state-of-the-art audio generation models like MusicGen and AudioGen. While these models excel at producing high-quality audio, they often fail to capture specific details in complex text prompts—such as the exact number of instruments or specific sound events. RSR improves controllability by ensuring that the semantic relationships between generated audio samples mirror the relationships between their corresponding text descriptions.

Methodology

The core of the proposed approach is a novel loss function that operates on batches of data rather than individual samples. Instead of a direct point-to-point mapping between audio and text (common in contrastive learning), RSR focuses on relational similarity.

  1. Similarity Matrices: For each training batch, the model computes two similarity matrices: one for the text prompts and one for the generated audio representations.
  2. Alignment Loss: The RSR loss minimizes the discrepancy between these two matrices. If two text prompts are semantically similar, the model is penalized if the resulting audio representations are not similarly related.
  3. Strategic Integration: RSR is applied during the Classifier-Free Guidance (CFG) phase. This forces the model to maintain a robust internal semantic structure even when explicit text guidance is occasionally removed during training, preventing the model from simply "copying" text features.
  4. Pooling Strategy: The researchers found that max pooling is more effective than average pooling for condensing sequences into vectors, as it better captures salient "peaks" in audio and text.
The RSR architecture integrates representation regularization into the transformer-based audio generation pipeline.

Experimental Setup

The models were trained on extensive datasets:

  • Music: 20,000 hours of licensed music from internal Meta sources, Shutterstock, and Pond5.
  • Sound Effects (SFX): 4,000 hours of audio from AudioSet, BBC Sound Effects, AudioCaps, and others.
  • Evaluation: Performance was measured using the MusicCaps and AudioCaps benchmarks, focusing on Frechet Audio Distance (FAD), KL divergence, and CLAP alignment scores.

Key Results

The implementation of RSR led to significant improvements in both objective and subjective performance. Notably, the 300M parameter model utilizing RSR was able to outperform the significantly larger 1.5B parameter MusicGen model in key metrics.

Objective performance metrics comparing RSR-enhanced models against state-of-the-art baselines.

Beyond objective scores, human evaluators showed a strong preference for RSR-enabled models in blind A/B tests. This preference was particularly pronounced in sound effect generation, where the RSR model was preferred 67% of the time over the baseline. Evaluators noted that the RSR model followed prompts more accurately, correctly rendering specific details like the number of "ping pong bounces" that the baseline often missed.

Conclusion and Future Work

Representation Similarity Regularization provides a powerful yet computationally manageable way to enhance the controllability of generative audio models. By focusing on relational consistency, RSR mitigates prompt neglect and improves semantic alignment. While the impact is most visible in sound effects due to their concrete nature, the method also shows promise for music generation. Future research will explore scaling RSR to multi-billion parameter models and further refining the regularization techniques for abstract musical concepts.

Summary updated

Abstract

This paper presents an innovative approach to enhance control over audio generation by emphasizing the alignment between audio and text representations during model training. In the context of language model-based audio generation, the model leverages input from both textual and audio token representations to predict subsequent audio tokens. However, the current configuration lacks explicit regularization to ensure the alignment between the chosen text representation and the language model's predictions. Our proposal involves the incorporation of audio and text representation regularization, particularly during the classifier-free guidance (CFG) phase, where the text condition is excluded from cross attention during language model training. The aim of this proposed representation regularization is to minimize discrepancies in audio and text similarity compared to other samples within the same training batch. Experimental results on both music and audio generation tasks demonstrate that our proposed methods lead to improvements in objective metrics for both audio and music generation, as well as an enhancement in the human perception for audio generation.
BibTeX
@misc{shi2023enhance,
  author = {Yangyang Shi and Gaël Le Lan and Varun Nagaraja and Zhaoheng Ni and Xinhao Mei and Ernie Chang and Forrest N. Iandola and Yang Liu and Vikas Chandra},
  title = {{Enhance audio generation controllability through representation similarity regularization}},
  year = {2023},
  eprint = {2309.08773},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2309.08773},
}
Figure