Using structure patterns of temporal and spectral feature in audio similarity measure

Rui Cai, Lie Lu, HongJiang ZhangACM Multimedia 2003 · DOI

TL;DR

This paper introduces "structure patterns" – descriptive models of temporal and spectral audio characteristics – to improve audio similarity measurement beyond traditional statistical methods. By capturing the "shape" of audio features, these patterns significantly enhance the performance of content-based audio retrieval systems.

What Problem Does This Paper Address?

Current audio similarity measures often rely on statistical summaries (mean, variance) of temporal and spectral features extracted from audio frames. While effective to a degree, these methods can be inaccurate because they overlook the detailed temporal evolution and spectral structure within audio clips. For instance, two very different sounds might have similar statistical profiles, while two instances of the same sound (like a jet plane) might appear statistically dissimilar due to variations in energy distribution or spectral content. This paper aims to bridge this gap by proposing a more nuanced approach that captures the inherent "structure" of audio features.

Methods

The authors propose three types of "structure patterns":

  • Energy Envelope Pattern: This pattern captures the temporal evolution of a sound's energy. They fit polynomial curves to normalized energy envelopes and use k-means clustering to identify five representative patterns (e.g., rapid attack/decay, long sustain). Each audio clip is then assigned one of these patterns.
  • Harmonicity Pattern: This pattern describes the harmonic characteristics within different frequency sub-bands of an audio signal. They analyze narrow-band spectrograms, thresholding them to focus on significant spectral content. For six defined frequency sub-bands, they classify the harmonic status as "silence," "noise," "semi-harmonic," or "harmonic" based on spectral energy and peak detection.
  • Pitch Contour Pattern: This pattern represents the shape of the fundamental frequency (pitch) over time. After extracting pitch contours, they divide them into segments and assign semantic labels (absence, increase, decrease, sustain) to describe the pitch's evolution.

These structure patterns are then integrated with traditional statistical features into a unified similarity measure. The overall distance between two audio segments is a weighted combination of the distance between their statistical features and the distances between their respective structure patterns (energy envelope, harmonicity, and pitch contour). For longer audio clips, they segment the audio based on energy contour and compute the minimum distance between segments of the two clips.

Datasets

The paper uses a custom database of approximately 600 audio clips. These clips vary in duration (less than 1 to 30 seconds) and encompass a wide range of sound types, including animals, machines, vehicles, human sounds, and weapons. The sample rate for all sounds is 22050 Hz. The paper does not introduce a new benchmark dataset but evaluates its methods on this diverse collection.

Key Insights and Ideas

  • Structure Matters: The core idea is that the shape or pattern of audio features, not just their average statistical values, is crucial for accurate similarity perception and measurement.
  • Complementary Information: Structure patterns provide complementary information to statistical features, capturing nuances that statistics alone miss.
  • Human Perception Alignment: The proposed patterns (energy envelope, harmonicity, pitch) are motivated by psychophysical research suggesting their importance in human auditory perception of sound objects.
  • Effective for Sound Effects: The approach shows particular promise for distinguishing and retrieving sound effects, which often have distinct temporal and spectral structures.

Key Results

Experiments were conducted using a content-based audio retrieval system. The results demonstrate significant improvements when structure patterns are integrated:

  • Improved Retrieval Performance: For a "jet plane" sound query, integrating spectral and temporal structure patterns improved the ranks of target clips from initial positions like 3, 8, and 11 to 1, 3, and 5, respectively.
  • General Performance Gains: When tested with a broader set of queries (sheep, gun, racing car, applause, etc.), integrating structure patterns led to substantial increases in both recall and precision. For example, in the top 30 retrieved results, recall increased from approximately 40% to 70%, and precision also saw a 60% increase.
  • Distinct Improvement: The authors conclude that the proposed structure patterns provide a distinct improvement in audio similarity measurement for sound effects.

Limitations and Future Work

Limitations: * The paper acknowledges that the current approach primarily focuses on single "gestalts" (distinct sound events) and discusses a segmentation strategy for longer, multi-gestalt sounds, but more advanced segmentation might be needed. * The choice of polynomial degree for energy envelope fitting (n=10) and the thresholds for harmonicity detection were empirically chosen and might require further optimization.

Future Work: The authors suggest two main directions for future research:

  1. Design More Patterns: Develop additional "effective and representative structure patterns" that are important and relevant to auditory perception.
  2. Better Integration: Explore better ways to integrate these structure patterns into similarity measures, potentially with different weighting schemes or alongside traditional statistical features.

Summary updated

BibTeX
@inproceedings{cai2003using,
  author = {Rui Cai and Lie Lu and HongJiang Zhang},
  title = {{Using structure patterns of temporal and spectral feature in audio similarity measure}},
  booktitle = {ACM Multimedia},
  year = {2003},
  doi = {10.1145/957013.957056},
}
Figure