Multipitch estimation of quasi-harmonic sounds in colored noise

Valentin Emiya, Roland Badeau, Bertrand David2007

TL;DR

This paper introduces a novel multipitch estimation method for quasi-harmonic sounds embedded in colored noise. By modeling the sound as a sum of sinusoids with an autoregressive spectral envelope and a moving-average noise, the authors propose a Weighted Maximum Likelihood (WML) approach to jointly estimate multiple fundamental frequencies. This method shows promising results, particularly for piano tones, by effectively separating musical notes from background noise.

What Problem Does This Paper Address?

Accurately identifying multiple simultaneous musical notes (multipitch estimation) is a fundamental challenge in audio signal processing, crucial for applications like automatic music transcription and analysis. When musical notes are present, their harmonic overtones can overlap, creating spectral interference that makes it difficult to distinguish individual notes. This problem is further compounded by the presence of background noise, especially colored noise, which can obscure the harmonic structure of the notes. Existing methods often struggle with this spectral overlap and noise, leading to errors in pitch estimation. This paper aims to tackle this by developing a robust estimator that can disentangle multiple quasi-harmonic sounds from colored noise.

Methods

The core of the proposed method is a Weighted Maximum Likelihood (WML) principle. The authors model the audio signal as a sum of sinusoidal components (representing musical notes) and a colored noise. Specifically:

  • Sinusoidal Model: Each note is assumed to have a spectral envelope modeled by an autoregressive (AR) filter. This captures the decaying amplitude of overtones.
  • Noise Model: The background noise is modeled as a Moving-Average (MA) process, characterized by an MA filter.
  • Likelihood Maximization: The WML principle involves partitioning the signal's frequency bins into two sets: one associated with the sinusoidal components (H) and one with the noise (N). The goal is to find the fundamental frequencies (F0s) that, when used to construct the AR model for the notes and the MA model for the noise, maximally "whiten" both the sinusoidal and noise sub-spectra. This means making the spectral flatness of both components as high as possible.
  • Monopitch to Multipitch: A monopitch estimator is first derived, which finds the F0 that best flattens both spectra. This is then extended to a multipitch estimator by jointly estimating multiple F0s.
  • Handling Inharmonicity: For instruments like the piano, where overtones are not perfectly harmonic, the method incorporates a specific inharmonicity law to define the expected overtone frequencies.
  • Practical Adaptations: To handle real-world audio, the method includes practical adaptations such as using a Hann window for analysis to mitigate spectral leakage and employing parabolic interpolation in the frequency domain to estimate overtone amplitudes more accurately.

Datasets

The paper was evaluated on:

  • A database of approximately 540 isolated piano tones from the RWC database.
  • Randomly generated chords created by virtual piano software. Specifically, around 600 two-note chords and 600 three-note chords were used.

The authors note that these test conditions are similar to those used in competitor systems, allowing for direct comparison.

Key Insights and Ideas

  • Joint Spectral Flattening: The core insight is that by jointly maximizing the spectral flatness of both the signal's harmonic components (using an AR model) and the residual noise (using an MA model), one can effectively isolate and estimate multiple fundamental frequencies. This is a powerful way to leverage the distinct statistical properties of musical notes and colored noise.
  • AR/MA Model Synergy: The combination of AR models for the harmonic structure and MA models for the noise is crucial. The AR model is good at capturing the decaying nature of overtones, while the MA model is effective at modeling the residual colored noise after the signal components have been accounted for.
  • Inharmonicity as a Constraint: Recognizing and modeling the inharmonicity of instruments like the piano is essential for accurate estimation, as it deviates from the idealized harmonic series.
  • Practical Robustness: The adaptations for real-world audio (windowing, amplitude interpolation) demonstrate that the theoretical framework can be made robust enough to handle the complexities of actual musical sounds.

Key Results

  • Monopitch Performance: The monopitch estimator was shown to effectively flatten both the noise and sinusoidal sub-spectra, leading to accurate F0 estimation. Figures 1-4 illustrate how the method separates components and whitens residuals.
  • Multipitch Performance: The multipitch estimator achieved competitive error rates compared to state-of-the-art methods.
  • Monophony: 2.0% error rate.
  • Two-note polyphony: 7.5% error rate.
  • Three-note polyphony: 23.9% error rate.
  • Comparison to State-of-the-Art: The error rates for polyphony one and two were particularly competitive, with the method performing comparably or better than existing systems. For polyphony three, the error rate was within the expected range for such complex scenarios.
  • Qualitative Analysis: Figure 6 visually demonstrates the estimation results for random chords, showing the ability to correctly identify notes (crosses) and distinguish them from false positives and false negatives.

Limitations and Future Work

  • Computational Cost: The paper acknowledges that a direct implementation of the joint estimation for multipitch can be computationally very expensive due to the combinatorial nature of evaluating all possible note combinations.
  • Overlap Between Notes: The current method has limitations in managing significant overlap between the spectra of closely spaced notes.
  • Model Limitations: The authors suggest future work could involve improving the AR model for spectral envelopes and exploring more sophisticated noise models.
  • Non-Stationarity: The paper notes that the non-stationary nature of real audio signals poses a challenge for separating components and residual signals, suggesting this needs further attention.
  • Future Directions: Future work will focus on managing note spectral overlap, enhancing the spectral envelope model, and reducing computational complexity to make the estimator more efficient and practical for real-time applications.

Summary updated

Abstract

This paper proposes a new multipitch estimator based on a likelihood maximization principle. For each tone, a sinusoidal model is assumed with a colored, Moving-Average, background noise and an autoregressive spectral envelope for the overtones. A monopitch estimator is derived following a Weighted Maximum Likelihood principle and leads to find the fundamental frequency (F0) which jointly maximally flattens the noise spectrum and the sinusoidal spectrum. The multipitch estimator is obtained by extending the method for jointly estimating multiple F0’s. An application to piano tones is presented, which takes into account the inharmonicity of the overtone series for this instrument. 1.
BibTeX
@misc{emiya2007multipitch,
  author = {Valentin Emiya and Roland Badeau and Bertrand David},
  title = {{Multipitch estimation of quasi-harmonic sounds in colored noise}},
  year = {2007},
}
Figure