A Qualitative Assessment of Measures for the Evaluation of a Cover Song Identification System

Joan SerràISMIR 2007 · OpenReview

Abstract

The identification of musical cover versions presents a significant challenge for Information Retrieval (IR) systems due to the extreme imbalance between relevant and non-relevant documents. This paper evaluates various performance measures to determine which best reflect system quality in the Cover Song Identification (CSI) task. By analyzing six synthetic answer sets representing distinct retrieval behaviors, we demonstrate that standard metrics like Accuracy and Precision are inadequate. Instead, we propose $bpref^*$ and Normalized Lift Curves as superior tools for assessing both recall and ranking effectiveness.

Introduction

Identifying a "cover song"—a new rendition of a previously recorded track—is a unique challenge in Music Information Retrieval (MIR). Unlike simple audio fingerprinting, which looks for exact matches, CSI systems must recognize the same underlying musical "essence" despite changes in tempo, key, instrumentation, or genre.

The primary difficulty lies not just in the identification itself, but in how success is measured. In a database of thousands of songs, a specific query might only have two or three valid covers. This creates an extremely "skewed" dataset where 99.9% of the documents are irrelevant. Standard metrics used in machine learning, such as Accuracy, become meaningless because a system could achieve 99.9% accuracy simply by guessing "not a cover" every single time.

Methodology

To evaluate which metrics actually reflect system quality, a qualitative assessment was performed using a test framework of six synthetic answer sets ($A_1$ through $A_6$). These sets represent hypothetical system behaviors:

  • $A_1$: Finds the only existing cover immediately.
  • $A_2$ & $A_3$: Find several covers but rank them differently.
  • $A_4$ & $A_5$: Find many covers but buried deeper in the results.
  • $A_6$: Fails to find any covers.

These sets were applied to a database of 2,054 songs with an average of 4.24 covers per song. A battery of standard IR measures was then applied to these controlled scenarios to observe which aligned with human intuition of "quality."

Evaluating Metric Performance

The analysis revealed that many traditional metrics fail to distinguish between good and mediocre results in the CSI domain. As shown in the performance table below, measures like Specificity and Fallout Rate show almost no variation between systems due to the massive number of True Negatives (TN), making them useless for algorithm tuning.

Table 1: Comparison of various IR measures across the six synthetic answer sets.

While Average Precision (AP) is a staple in IR, it does not adequately account for whether all possible covers in the database were retrieved. The study highlights Binary Preference (bpref) as a more effective alternative. Specifically, the proposed $bpref^*$ (a modified version of $bpref$) proved most capable of distinguishing between the synthetic sets by rewarding both high-ranking covers and high recall, even when the number of relevant documents is very small.

Visualizing Performance

Beyond scalar metrics, the way performance is visualized also matters. While Precision-Recall curves are standard, they can be difficult to interpret in highly skewed datasets. The study found that Normalized Lift Curves—which plot the percentage of positive examples found against the ratio of the answer set explored—provided a much clearer picture of system performance.

Figure 1: Normalized Lift Curves for the synthetic sets, showing the percentage of positive examples found vs. the percentage of the database explored.

As seen in Figure 1, the Lift Curves clearly differentiate the rapid success of $A_1$ and $A_2$ from the delayed or non-existent success of $A_5$ and $A_6$. This visualization offers an intuitive understanding of how quickly a system "lifts" relevant results to the top of the list.

Conclusion

Evaluation measures in MIR are not "one size fits all." For researchers in CSI, the choice of metric is critical to ensure that algorithms are being rewarded for the right behaviors. This study concludes that $bpref^*$ and Normalized Lift Curves are the most robust tools for this task, as they remain discriminative despite extreme data skewness and prioritize the ranking of relevant results.

Summary updated

BibTeX
@inproceedings{serra2007qualitative,
  author = {Joan Serrà},
  title = {{A Qualitative Assessment of Measures for the Evaluation of a Cover Song Identification System}},
  booktitle = {ISMIR},
  year = {2007},
  url = {https://openreview.net/forum?id=wgzWrqLu08},
}
Figure