From HEAR to GEAR: Generative Evaluation of Audio Representations

Vincent Lostanlen, Lingyao Yan, Simon X. YangHAL 2023

TL;DR

This paper introduces GEAR, a novel method for evaluating audio representations by generating synthetic sounds with controlled acoustic properties and visualizing how well different audio models can disentangle these properties. GEAR offers a complementary, more interpretable perspective to the HEAR benchmark, aiming to improve our understanding of what audio representations truly learn.

What Problem Does This Paper Address?

The field of machine listening, or audio content analysis, has seen rapid advancements, largely driven by deep learning. However, evaluating these systems presents challenges. The HEAR benchmark was introduced to provide a standardized, holistic evaluation of audio representations across many tasks. While HEAR is valuable for comparing models on real-world tasks, it can be difficult to interpret why certain representations perform better than others. It's hard to tell if a model's success is due to its inherent ability to capture fundamental audio properties or simply due to good hyperparameter tuning for specific tasks. This paper addresses this "black box" problem by proposing GEAR (Generative Evaluation of Audio Representations), a method focused on interpretability and understanding the disentanglement capabilities of audio representations.

Methods

GEAR's core idea is to use a generative model to create synthetic audio signals with controlled, independent factors of variability. These synthetic sounds are then passed through various audio representations (acting as feature extractors). The resulting embeddings are visualized using manifold learning (specifically Isomap) to create a 3D scatter plot. The points in this plot are colored according to the ground truth synthesis parameters (e.g., pitch, spectral decay). The hypothesis is that a good audio representation should produce a "dense arrangement of points" where the colors change smoothly along orthogonal axes, indicating that the representation has successfully disentangled the underlying acoustic properties without supervision.

The paper also includes a quantitative evaluation using nearest-neighbor regression. This involves predicting the synthesis parameters from the embeddings and measuring the error.

Datasets

The paper introduces a new synthetic dataset for evaluation. This dataset is generated using an additive Fourier synthesis model, allowing for control over three key parameters:

  • f1: Fundamental frequency (pitch).
  • a: Fourier decay (spectral brightness).
  • r: Relative odd-to-even amplitude difference (related to resonator type, e.g., clarinet vs. flute).

This synthetic dataset consists of 2500 samples, with variations in these parameters. The paper does not use any real-world datasets for the core GEAR evaluation, as its focus is on controlled generative synthesis.

Key Insights and Ideas

  • GEAR as a "smoke test" for audio representations: The method acts as a visual diagnostic tool to quickly assess if an audio representation can disentangle basic acoustic properties.
  • Complementary to HEAR: GEAR's strength lies in its interpretability, addressing HEAR's weakness in explaining why models perform as they do.
  • Visualizing disentanglement: The Isomap visualization provides an intuitive way to see how well an embedding space preserves the underlying structure of the synthetic data. Smooth color gradients suggest good disentanglement.
  • Parametric synthesis is key: The ability to generate audio with controlled, independent parameters is crucial for this evaluation approach.
  • Not all representations are equal: Different audio representations exhibit vastly different topological structures in their embedding spaces when visualized with GEAR.

Key Results

  • Visual Inspection:
    • The logmelspec baseline is highly sensitive to pitch (f1), especially with bright spectra.
  • Open-L³ shows discontinuities in its pitch representation, indicating a loss of topological information.
  • BYOL and GURA show 2D manifolds primarily driven by pitch (f1) and spectral decay (a), with clarinet-like sounds (r=1) appearing as outliers.
  • CREPE is strongly sensitive to pitch (f1) and largely invariant to spectral envelope, as expected since it was trained for pitch estimation.
  • Wav2vec2 produces a 3D manifold, but its axes don't align with the original synthesis parameters.
  • YAMNet is highlighted as the best performer in the GEAR benchmark, producing a dense 3D manifold where synthesis parameters (f1, a, r) appear as smoothly changing over perpendicular directions.
  • Nearest-neighbor Regression: This quantitative evaluation was largely inconclusive, with most embeddings performing similarly for parameters 'a' and 'r'. Only the logmelspec baseline and CREPE showed high sensitivity to pitch (f1), as expected. The authors suggest this method might not be suitable for local neighborhood analysis and future work should focus on global invariance.

Limitations and Future Work

  • Lack of direct applicability: GEAR, in its current form, is not directly tied to real-world classification tasks, making it a "smoke test" rather than a direct performance measure.
  • Perceptual calibration: The synthetic parameters (f1, a, r) are not sampled according to a perceptually uniform scale. Future work should incorporate psychoacoustic knowledge (e.g., just-noticeable differences) to make the synthesizer more meaningful for human perception.
  • Euclidean nearest-neighbor search: The current method relies on Euclidean distances in feature space. Future work could explore aligning GEAR with HEAR's use of shallow neural networks by performing analysis at deeper representation levels, potentially involving neural architecture search.
  • Limited factors of variability: The current study only uses three low-level acoustic attributes. Expanding to more complex synthetic datasets (e.g., virtual drum shapes, text-to-speech) is a promising direction.
  • Automating interpretation: Visual inspection is subjective. Future work could involve topological data analysis (TDA) to automate the characterization of feature spaces.

Summary updated

Abstract

International audience
BibTeX
@inproceedings{lostanlen2023from,
  author = {Vincent Lostanlen and Lingyao Yan and Simon X. Yang},
  title = {{From HEAR to GEAR: Generative Evaluation of Audio Representations}},
  booktitle = {HAL},
  year = {2023},
}
Figure