Abstract
Traditional audio analysis pipelines often suffer from information loss during the transition between 1D waveforms and 2D spectrograms, particularly regarding phase information. This paper presents a unified probabilistic framework, Gaussian Time-Frequency Non-negative Matrix Factorization (GTF-NMF), which integrates time-frequency analysis and NMF into a single end-to-end model. By employing state-space Gaussian Processes and Power Expectation Propagation (EP), the proposed method enables high-fidelity audio processing tasks such as denoising and source separation while preserving phase and maintaining linear computational scaling. Experimental results demonstrate that the EP-based approach significantly outperforms standard linearization techniques like the Extended Kalman Filter (EKF) in recovering latent signals.
Introduction: The "Broken" Audio Pipeline
Standard audio processing typically follows a disjointed pipeline. A one-dimensional time-domain waveform is transformed into a two-dimensional magnitude spectrogram, a process that discards phase information. Subsequent analysis, such as Non-negative Matrix Factorization (NMF), is performed on this magnitude representation to identify spectral patterns. Reconstructing the audio then requires "guessing" the missing phase, often leading to audible artifacts and a loss of uncertainty information. Furthermore, modified spectrograms may not correspond to any valid 1D waveform, a problem known as the "consistency" issue.
While Gaussian Processes (GPs) provide a principled Bayesian framework for modeling signals with uncertainty, their $O(T^3)$ computational scaling makes them impractical for audio. A standard 16kHz audio track generates tens of thousands of data points per second, rendering traditional GP methods computationally infeasible.
The GTF-NMF Framework
To address these challenges, the authors introduce the Gaussian Time-Frequency NMF (GTF-NMF) model. This generative model treats the audio signal as a sum of frequency "carriers" (oscillators) modulated by temporal "modulators" (NMF components). Both carriers and modulators are modeled as Gaussian Processes.
In this framework, the "spectrogram" is not a fixed preprocessing step but a latent variable learned during joint inference. By operating directly on the waveform, the model naturally preserves phase information, leading to higher-fidelity reconstructions compared to magnitude-only methods.
Efficient Inference via State-Space GPs and Power EP
The primary technical hurdle is the non-linear interaction between modulators and carriers and the massive scale of audio data. The authors employ several strategies to make inference tractable:
- State-Space Representation: By reformulating GP priors as Stochastic Differential Equations (SDEs), the authors leverage Kalman filtering and smoothing techniques. This reduces the computational complexity from $O(T^3)$ to $O(T)$, enabling the processing of signals with over 96,000 data points.
- Power Expectation Propagation (EP): The non-linearity of the modulator-carrier product is too severe for standard tools like the Extended Kalman Filter (EKF). The authors utilize Power EP to approximate the complex posterior distribution. This iterative approach provides significantly more accurate recovery of latent signals than simple linearization.
- Infinite-Horizon GPs (IHGP): To manage memory bottlenecks associated with storing large covariance matrices over long durations, a steady-state IHGP approximation is used, reducing memory requirements from $O(TM^2)$ to $O(TM)$.
Experimental Results
The model was rigorously evaluated across simulated data, musical instrument recordings, speech, and piano mixtures.
Missing Data Imputation
In tasks where 20ms segments of audio were removed, the EP-based method achieved a Signal-to-Noise Ratio (SNR) of 8.087 dB. This performance substantially outperformed the EKF baseline (3.735 dB) and traditional matching pursuit algorithms (5.232 dB), highlighting the necessity of the EP framework for handling strong non-linearities.
Denoising and Source Separation
The framework effectively removed Gaussian noise from speech samples. While specialized "black-box" deep learning models may achieve higher scores on specific benchmarks, GTF-NMF offers a flexible, general-purpose probabilistic framework that does not require massive training datasets. Furthermore, the model successfully separated three overlapping piano notes (C, E, G) from a single mixture, demonstrating its ability to handle high-dimensional state spaces ($M=123$) over extended durations.
Limitations and Future Outlook
Despite its successes, the GTF-NMF model faces several challenges. The computational cost of 9th-order sigma-point methods used in EP scales poorly with the number of NMF components, limiting its use for complex polyphonic music. Additionally, the model exhibits sensitivity to hyperparameters, as the "non-identifiability" of the carrier-modulator product makes learning exact filter frequencies difficult. Future research will focus on more efficient parameter learning schemes and further computational optimizations to enable real-time applications.
Summary updated