High-Level Audio Features: Distributed Extraction and Similarity Search

François Deliège, Bee Yong Chua, Torben Bach PedersenISMIR 2008 · OpenReview

Abstract

Extracting high-level audio features from large-scale music datasets is often hindered by significant computational requirements and legal restrictions regarding copyright. This paper introduces the iSound Music Warehouse (iSoundMW), a distributed computing framework designed to address these challenges. iSoundMW utilizes a novel two-step extraction process that segments audio files to comply with fair use policies and employs an optimized database architecture to facilitate efficient, user-weighted similarity searches. Experimental results demonstrate that the system scales effectively, supporting thousands of concurrent clients and achieving sub-100ms query times for similarity searches.

Introduction

Music Information Retrieval (MIR) relies on the extraction of high-level features—such as mood, tempo, and pitch—to power modern recommendation engines. However, researchers face a dual bottleneck: the heavy computational load of extraction algorithms and the legal inability to distribute full copyrighted audio files across public computing grids. These constraints often limit research to small, private datasets, hindering the generalizability of MIR findings. Furthermore, querying large feature sets for similarity is complex because musical perception often violates the geometric assumptions (like the triangle inequality) used in standard database indexing.

The iSoundMW Architecture

To overcome these barriers, the authors developed iSoundMW, a client-server architecture designed for high-throughput feature extraction and retrieval.

The iSoundMW client-server architecture for distributed music processing.

As shown in the architecture diagram, the system is divided into a Server Side, which manages the audio collection and job scheduling, and a Client Side, which performs the actual computation. The Application Server hosts a Music Search Engine, a Job Manager, and a Segment Handler, coordinating with a Music Information Database to store extracted metadata.

Distributed Feature Extraction

The core innovation of iSoundMW is its two-step distributed extraction process, which ensures copyright compliance while distributing the workload:

  1. Segment Extraction: The server divides MP3 files into non-overlapping 5-second segments. These short, semi-randomly assigned snippets are sent to remote "Extractor" clients. This approach falls under "fair use," as the full song cannot be reconstructed from the segments, and it normalizes the computational load across nodes.
  2. Feature Aggregation: Once base features are extracted from segments, the server redistributes these data points to "Aggregator" clients. These clients synthesize the segment-level data into comprehensive song-level features.
Step-by-step interaction between the application server, extractor, and aggregator.

This workflow, managed by the central Job Manager, allows the system to handle thousands of job requests per second, effectively turning copyright compliance into a load-balancing strategy.

Beyond extraction, iSoundMW optimizes how similarity queries are executed. Features are categorized into two types:

  • Absolute Features: Simple metrics like tempo that can be compared on-the-fly using an absolute similarity table (abssim).
  • Relative Features: Complex features like timbre that require pre-computed comparisons stored in a relative similarity table (relsim).

To handle the non-metric nature of music similarity, the system avoids traditional indexing. Instead, it uses partial indexes and compressed arrays to filter out highly dissimilar songs early in the query process. This reduces disk I/O and speeds up the expensive database joins required for Top-K similarity searches.

Experimental Results

The system was evaluated using a dataset of 41,446 songs, processed into over 2.2 million 5-second segments. Key performance metrics include:

  • Scalability: The job manager handles up to 1,766 requests per second.
  • Throughput: The server supports 87 clients per second during active segmentation; without the CPU-intensive segmentation step, this capacity increases to 9,000 clients.
  • Search Efficiency: A Top-10 similarity search takes approximately 575 ms. Optimization techniques (range constraints and compressed arrays) reduced query times from 130 ms to just 15 ms in specific test cases.

Conclusion

iSoundMW provides a robust framework for large-scale MIR research by balancing computational efficiency with legal compliance. Future work will focus on scaling the system with multiple segment handlers to support diverse music collections and integrating user feedback loops to further refine recommendation accuracy.

Summary updated

BibTeX
@inproceedings{deliege2008high,
  author = {François Deliège and Bee Yong Chua and Torben Bach Pedersen},
  title = {{High-Level Audio Features: Distributed Extraction and Similarity Search}},
  booktitle = {ISMIR},
  year = {2008},
  url = {https://openreview.net/forum?id=BeTQIKsAZR},
}
Figure