Sparsity and persistence in time-frequency sound representations

Matthieu Kowalski, Bruno TorrésaniProceedings of SPIE, the International Society for Optical Engineering/Proceedings of SPIE 2009 · DOI

TL;DR

This paper introduces a novel framework for analyzing audio signals by exploiting their inherent sparsity and persistence in the time-frequency domain. The authors propose hierarchical probabilistic models and algorithms to decompose audio into distinct layers (like tonal and transient components), offering a more structured and interpretable representation than traditional methods.

What Problem Does This Paper Address?

Audio signals, particularly music, are known to be well-represented in the time-frequency domain. Two key characteristics are:

  1. Sparsity: A small number of time-frequency coefficients can accurately approximate the signal.
  2. Persistence: Significant coefficients tend to appear in clusters rather than being isolated.

Traditional signal processing methods often struggle to fully leverage these properties, especially when dealing with complex audio signals that can be decomposed into multiple, interacting components (e.g., sustained tones vs. sharp transients). The challenge is to develop models and algorithms that can effectively identify and separate these components, leading to richer signal representations for applications like audio analysis, coding, and transformation. This paper aims to address this by proposing a hierarchical probabilistic approach that explicitly models both sparsity and the structured persistence of coefficients.

Methods

The core of the paper lies in developing hierarchical probabilistic models for audio signals. The approach can be summarized as follows:

  • Dictionary Choice: The authors utilize time-frequency representations, specifically focusing on unions of MDCT (Modified Discrete Cosine Transform) bases or Gabor frames. These dictionaries are chosen to offer different time-frequency resolutions, allowing for the potential separation of tonal (frequency-selective) and transient (time-selective) components.
  • Hierarchical Modeling: Coefficients within the chosen dictionary are grouped. Each group is then modeled as a random vector whose distribution is governed by a hidden state. This creates a hierarchical structure where higher-level states influence the properties of lower-level coefficients.
  • Hidden State Estimation: The paper focuses on estimating these hidden states from the analysis coefficients (obtained by projecting the signal onto the dictionary atoms). This is a crucial step for identifying which coefficients belong to which layer or component.
  • Algorithms: The authors propose algorithms for parameter inference and hidden state estimation. These algorithms are based on Expectation-Maximization (EM) or Classification Expectation-Maximization (CEM) principles, iteratively refining estimates of model parameters and hidden states. They introduce adaptive thresholding rules, which are coefficient-dependent, to identify significant coefficients and thus estimate the significance map.
  • Correlated Coefficients: A significant extension is the development of schemes that account for correlations between synthesis coefficients within groups, particularly for structured dictionaries like unions of bases. This leads to more complex models but allows for a more accurate representation of structured audio components.

Datasets

The paper uses a musical excerpt from Shusheela Raman's "mamavatu" as the primary dataset for numerical evaluation. The signal is sampled at 44.1 KHz and is approximately 12 seconds long. No new benchmarks are introduced; the focus is on applying the proposed methods to a standard audio signal.

Key Insights and Ideas

  • Exploiting Persistence: The key innovation is moving beyond simple sparsity to explicitly model the persistence or clustering of time-frequency coefficients. This allows for the identification of structured components like tonal and transient sounds, which are not isolated but tend to form patterns.
  • Hierarchical Structure for Audio: The hierarchical probabilistic models provide a natural way to represent the layered nature of audio signals. By assigning hidden states to groups of coefficients, the model can distinguish between different types of sound events.
  • Analysis Coefficients as Key: The paper emphasizes the importance of using analysis coefficients (rather than synthesis coefficients) for estimating hidden states. This is a practical approach that avoids the non-uniqueness issues often associated with signal synthesis.
  • Adaptive Thresholding: The development of coefficient-dependent adaptive thresholds is a novel algorithmic contribution. These thresholds dynamically adjust based on the signal's properties, leading to more accurate identification of significant coefficients and thus better layer separation.
  • Structured Dictionaries for Layer Separation: The use of dictionaries composed of unions of bases with different time-frequency resolutions (e.g., one for tonal, one for transient) is a practical strategy for enabling the separation of these distinct audio components.

Key Results

  • Successful Layer Separation: The numerical results demonstrate that the proposed methods, particularly the "vector" model and its associated algorithms, can achieve a better separation of tonal and transient layers compared to simpler Bernoulli models. The estimated significance maps show sharper distinctions, with the transient layer being less contaminated by low-frequency tonal components.
  • Improved Representation: The method provides a more structured and interpretable representation of audio signals, allowing for the identification of distinct signal components.
  • Correlation Handling: The algorithms developed for correlated coefficients show promise in capturing the underlying structure of audio components, as evidenced by the correlation matrices of the estimated layers. The transient layer's correlation matrix is shown to be sparser, reflecting its more localized nature.

Limitations and Future Work

  • Computational Cost: The paper acknowledges that CEM algorithms can be computationally expensive, especially for updating the "gamma weights" (related to coefficient variances) at each iteration. This necessitates efficient storage of the dictionary's Gram matrix, which can be demanding for long signals.
  • Mean-Field Approximation: The numerical results primarily utilize a "mean-field" version of the algorithm, which offers computational efficiency but may be less optimal than full CEM.
  • Tonal Modeling: The authors note that the current procedure is particularly relevant for transient modeling but could be improved for tonal modeling. Specifically, time-independent significance maps for tonals are considered a limitation, suggesting the need for incorporating time-varying aspects into the tonal layer modeling.
  • Future Directions: The paper suggests incorporating an extra time scale into the algorithm for more dynamic tonal modeling and for updating model parameters over time. This would lead to a more comprehensive and adaptive audio analysis system.

Summary updated

Abstract

It is a well known fact that the time-frequency domain is very well adapted for representing audio signals. The main two features of time-frequency representations of many classes of audio signals are sparsity (signals are generally well approximated using a small number of coefficients) and persistence (significant coefficients are not isolated, and tend to form clusters). This contribution presents signal approximation algorithms that exploit these properties, in the framework of hierarchical probabilistic models. Given a time-frequency frame (i.e. a Gabor frame, or a union of several Gabor frames or time-frequency bases), coefficients are first gathered into groups. A group of coefficients is then modeled as a random vector, whose distribution is governed by a hidden state associated with the group. Algorithms for parameter inference and hidden state estimation from analysis coefficients are described. The role of the chosen dictionary, and more particularly its structure, is also investigated. The proposed approach bears some resemblance with variational approaches previously proposed by the authors (in particular the variational approach exploiting mixed norms based regularization terms). In the framework of audio signal applications, the time-frequency frame under consideration is a union of two MDCT bases or two Gabor frames, in order to generate estimates for tonal and transient layers. Groups corresponding to tonal (resp. transient) coefficients are constant frequency (resp. constant time) time-frequency coefficients of a frequency-selective (resp. time-selective) MDCT basis or Gabor frame.
BibTeX
@inproceedings{kowalski2009sparsity,
  author = {Matthieu Kowalski and Bruno Torrésani},
  title = {{Sparsity and persistence in time-frequency sound representations}},
  booktitle = {Proceedings of SPIE, the International Society for Optical Engineering/Proceedings of SPIE},
  year = {2009},
  doi = {10.1117/12.825220},
}
Figure