Estimation With Low-Rank Time-Frequency Synthesis Models

Cédric Févotte, Matthieu KowalskiIEEE TSP 2018 · DOI · OpenReview

Abstract

State-of-the-art signal decomposition often relies on low-rank factorization of time-frequency (t-f) transforms, most notably Nonnegative Matrix Factorization (NMF) of spectrograms. However, NMF is primarily an analysis-based approach that discards phase information. This paper proposes Low-Rank Time-Frequency Synthesis (LRTFS), a generative modeling paradigm that imposes low-rankness directly on the synthesis coefficients of a signal. By bridging t-f synthesis modeling with traditional NMF, LRTFS enables more robust signal recovery, handles phase implicitly, and excels in applications like denoising and compressive sensing.

Introduction

In audio signal processing, NMF has become a standard tool for source separation and feature extraction. Despite its success, standard NMF operates on the squared magnitude of the Short-Time Fourier Transform (STFT), treating it as an "analysis" step. This approach ignores the underlying phase of the signal, which must be heuristically reconstructed. Conversely, traditional "synthesis" models, such as sparse coding, focus on the raw time-domain signal but typically rely on sparsity priors. While sparsity is useful, it fails to capture the inherent low-rank structure of audio—where specific spectral patterns repeat over time.

LRTFS addresses these gaps by applying a low-rank prior to the synthesis coefficients within a generative framework. This allows for a more principled treatment of the signal's structure and phase, facilitating superior performance in inverse problems.

Methodology

The LRTFS framework models a signal $x$ as: $$x = \Phi\alpha + e$$ where $\Phi$ is a time-frequency dictionary (e.g., a Gabor frame), $\alpha$ are the synthesis coefficients, and $e$ represents additive white noise.

Low-Rank Prior

Unlike sparse models, LRTFS assumes that the variance of the coefficients $\alpha$ follows a low-rank structure: $$|\alpha_{fn}|^2 \approx [WH]_{fn}$$ Here, $W$ represents the spectral templates (dictionary of patterns) and $H$ represents the temporal activations, mirroring the structure of NMF.

Estimation and Algorithms

The authors employ Maximum Joint Likelihood Estimation (MJLE) to simultaneously estimate the coefficients $\alpha$ and the NMF factors $W$ and $H$. The optimization is handled via:

  1. Majorization-Minimization (MM): Used for updating $W$ and $H$ based on Itakura-Saito NMF principles.
  2. Iterative Shrinkage Algorithm (ISA): A fast, proximal gradient method used to update the synthesis coefficients $\alpha$.

This generative approach can be extended to multi-layer models, allowing the separation of distinct signal components, such as "tonal" parts (captured by low-rank structures) and "transient" parts (captured by sparse structures).

Comparison of signal decomposition between IS-NMF and the proposed LRTFS on a piano recording.

Experimental Results

The model was evaluated on audio datasets, including a piano recording and a musical excerpt ("Mamavatu").

Signal Decomposition and Denoising

As shown in Figure 1, LRTFS demonstrates a significant advantage over standard IS-NMF in decomposing a noisy piano signal. While IS-NMF often splits single notes into multiple components and fails to isolate transient "hammer attacks," LRTFS successfully recovers clean, distinct components. Because LRTFS models the raw signal, it maintains phase consistency, leading to more natural-sounding reconstructions.

Compressive Sensing (CS)

A key finding is the application of low-rankness to Compressive Sensing. Traditional CS relies on signal sparsity, but the authors show that the repetitive spectral structure of audio makes low-rankness a more powerful prior. In tests recovering signals from as little as 1% to 10% of original samples, LRTFS outperformed sparsity-based methods (like $\ell_1$ regularization) by several decibels in output SNR.

Spectrogram of synthesis coefficients estimated by LRTFS, highlighting the recovered low-rank structure.

Discussion and Conclusion

LRTFS provides a robust generative alternative to analysis-based NMF. By modeling synthesis coefficients with a low-rank prior, it bridges the gap between structured matrix factorization and raw signal processing.

Key Advantages: * Implicit Phase Handling: Operates on the time-domain signal, avoiding the "phase problem" of magnitude-only NMF. * Structural Flexibility: Multi-layer extensions allow for the simultaneous modeling of tonal and transient components. * Superior CS Recovery: Demonstrates that low-rankness is a more effective prior than sparsity for musical audio.

Future work will focus on reducing the computational complexity of the joint estimation and incorporating temporal correlations into the variance model to better capture the dynamic evolution of sound.

Summary updated

Abstract

Many state-of-the-art signal decomposition techniques rely on a low-rank factorization of a time-frequency (t-f) transform. In particular, nonnegative matrix factorization (NMF) of the spectrogram has been considered in many audio applications. This is an analysis approach in the sense that the factorization is applied to the squared magnitude of the analysis coefficients returned by the t-f transform. In this paper, we instead propose a synthesis approach, where low-rankness is imposed to the synthesis coefficients of the data signal over a given t-f dictionary (such as a Gabor frame). As such, we offer a novel modeling paradigm that bridges t-f synthesis modeling and traditional analysis based NMF approaches. The proposed generative model allows us in turn to design more sophisticated multilayer representations that can efficiently capture diverse forms of structure. Additionally, the generative modeling allows us to exploit t-f low-rankness for compressive sensing. We present efficient iterative shrinkage algorithms to perform estimation in the proposed models and illustrate the capabilities of the new modeling paradigm over audio signal processing examples.
BibTeX
@article{fevotte2018estimation,
  author = {Cédric Févotte and Matthieu Kowalski},
  title = {{Estimation With Low-Rank Time-Frequency Synthesis Models}},
  journal = {IEEE Trans. Signal Process.},
  year = {2018},
  doi = {10.1109/tsp.2018.2844159},
}
Figure