Abstract
Music similarity is inherently multidimensional, yet traditional content-based retrieval systems often collapse these nuances into a single-axis metric. We present a deep metric learning framework that disentangles music embeddings into distinct, controllable subspaces representing genre, mood, instrumentation, and tempo. By utilizing Conditional Similarity Networks (CSN) and a novel "track regularization" technique, our model maintains song identity while allowing for specific attribute-based queries. Evaluation on the Million Song Dataset and human-annotated data shows that our joint disentangled model outperforms both traditional baselines and sets of specialized networks.
Introduction
In creative industries, such as video production, editors frequently encounter the "music replacement" problem: finding a track that matches the specific "feel" or tempo of a placeholder song while differing in other aspects like genre or instrumentation. Traditional text-based search is often limited by sparse metadata, and standard content-based models treat similarity as a binary or scalar property. This paper addresses the need for a multidimensional approach to music similarity, allowing users to specify the exact traits they wish to match.
Methodology
The proposed system leverages a deep metric learning approach adapted from Conditional Similarity Networks. The core innovation lies in how the embedding space is structured and trained.
Disentangled Embeddings and Masking
The model learns a 256-dimensional embedding space, which is partitioned into four disjoint 64-dimensional subspaces. Each subspace is dedicated to a specific musical attribute: genre, mood, instrumentation, and tempo. During training and inference, "masking functions" are applied to activate only the relevant subspace for a specific similarity task.
Triplet Loss and Track Regularization
The model is trained using triplet loss, which minimizes the distance between an anchor ($x_a$) and a positive sample ($x_p$) while maximizing the distance to a negative sample ($x_n$) within a masked subspace. To prevent the model from losing the overall identity of a track, the authors introduce track regularization. This involves sampling triplets where the anchor and positive samples are different segments of the same song, forcing the entire embedding to maintain track-specific consistency.
Experimental Setup
The model was trained using the Million Song Dataset (MSD), with ground-truth labels derived from Last.FM tags (categorized into genre, mood, and instrument) and algorithmic tempo estimates. To validate the model's alignment with human perception, the authors collected 39,440 human annotations, forming the dim-sim dataset.
Results
The disentangled model was compared against several baselines, including MFCC-based vector quantization and non-disentangled deep models.
Key findings include:
- Superiority of Disentanglement: The joint model trained on all dimensions simultaneously outperformed a set of specialized networks. This suggests that the model benefits from sharing low-level acoustic features across different tasks.
- Impact of Track Regularization: Adding track regularization improved the model's ability to maintain song identity without sacrificing performance on specific attributes.
- High Tempo Accuracy: The model demonstrated exceptional performance in capturing tempo, achieving an accuracy of 0.955 in sub-dimension testing.
- Human Alignment: In user studies, the proposed model achieved a consistency score of 0.792 with human ratings, significantly higher than the MFCC-VQ baseline (0.654).
Conclusion
This research demonstrates that disentangled multidimensional embeddings provide a more nuanced and controllable approach to music similarity than traditional methods. By separating musical traits into distinct subspaces, the model enables advanced retrieval scenarios, such as multi-query searches or attribute-weighted similarity. Future work will explore applying these techniques to other audio domains, such as voice and speech analysis.
Summary updated