Introduction
Meta AI researchers have introduced Representation Similarity Regularization (RSR), a training technique designed to address the "prompt neglect" problem in state-of-the-art audio generation models like MusicGen and AudioGen. While these models excel at producing high-quality audio, they often fail to capture specific details in complex text prompts—such as the exact number of instruments or specific sound events. RSR improves controllability by ensuring that the semantic relationships between generated audio samples mirror the relationships between their corresponding text descriptions.
Methodology
The core of the proposed approach is a novel loss function that operates on batches of data rather than individual samples. Instead of a direct point-to-point mapping between audio and text (common in contrastive learning), RSR focuses on relational similarity.
- Similarity Matrices: For each training batch, the model computes two similarity matrices: one for the text prompts and one for the generated audio representations.
- Alignment Loss: The RSR loss minimizes the discrepancy between these two matrices. If two text prompts are semantically similar, the model is penalized if the resulting audio representations are not similarly related.
- Strategic Integration: RSR is applied during the Classifier-Free Guidance (CFG) phase. This forces the model to maintain a robust internal semantic structure even when explicit text guidance is occasionally removed during training, preventing the model from simply "copying" text features.
- Pooling Strategy: The researchers found that max pooling is more effective than average pooling for condensing sequences into vectors, as it better captures salient "peaks" in audio and text.
Experimental Setup
The models were trained on extensive datasets:
- Music: 20,000 hours of licensed music from internal Meta sources, Shutterstock, and Pond5.
- Sound Effects (SFX): 4,000 hours of audio from AudioSet, BBC Sound Effects, AudioCaps, and others.
- Evaluation: Performance was measured using the MusicCaps and AudioCaps benchmarks, focusing on Frechet Audio Distance (FAD), KL divergence, and CLAP alignment scores.
Key Results
The implementation of RSR led to significant improvements in both objective and subjective performance. Notably, the 300M parameter model utilizing RSR was able to outperform the significantly larger 1.5B parameter MusicGen model in key metrics.
Beyond objective scores, human evaluators showed a strong preference for RSR-enabled models in blind A/B tests. This preference was particularly pronounced in sound effect generation, where the RSR model was preferred 67% of the time over the baseline. Evaluators noted that the RSR model followed prompts more accurately, correctly rendering specific details like the number of "ping pong bounces" that the baseline often missed.
Conclusion and Future Work
Representation Similarity Regularization provides a powerful yet computationally manageable way to enhance the controllability of generative audio models. By focusing on relational consistency, RSR mitigates prompt neglect and improves semantic alignment. While the impact is most visible in sound effects due to their concrete nature, the method also shows promise for music generation. Future research will explore scaling RSR to multi-billion parameter models and further refining the regularization techniques for abstract musical concepts.
Summary updated