Crepe: A Convolutional Representation for Pitch Estimation

Jong Wook Kim, Justin Salamon, Peter Li, Juan Pablo Bello

Abstract

Pitch estimation is a fundamental task in audio processing, traditionally dominated by hand-crafted digital signal processing (DSP) algorithms. This paper introduces CREPE (Convolutional Representation for Pitch Estimation), a deep learning-based pitch tracker that learns to estimate fundamental frequency ($f_0$) directly from raw audio waveforms. By utilizing a deep convolutional neural network, CREPE achieves state-of-the-art accuracy and demonstrates significantly improved robustness to noise compared to established methods like pYIN.

Introduction

Pitch estimation, or $f_0$ tracking, involves determining the fundamental frequency of a monophonic sound. Despite being a foundational task in audio engineering—used in speech analysis, music information retrieval, and more—it remains challenging due to complex timbres, rapid pitch fluctuations, and background noise. Traditional algorithms, such as the YIN algorithm or Autocorrelation Function (ACF), rely on human-designed heuristics that often fail in non-ideal acoustic environments. CREPE addresses these limitations by adopting a data-driven approach, learning pitch representations directly from raw audio data.

Methodology

The core of CREPE is a deep convolutional neural network (CNN) that operates directly on time-domain waveforms, bypassing the need for intermediate representations like spectrograms.

Architecture

The model consists of six convolutional layers followed by a fully connected output layer. It processes a 1024-sample window of audio at a 16 kHz sampling rate.

The CREPE architecture: a deep CNN with six convolutional layers processing raw audio samples to produce a 360-dimensional pitch representation.

Output Representation and Training

Instead of a single scalar value, CREPE outputs a 360-dimensional vector. Each bin corresponds to a specific pitch value in cents, spanning six octaves. During training, the target output is a Gaussian-blurred curve centered on the ground truth pitch. This spatial smoothing allows the model to learn the relationships between adjacent pitch bins and reduces the penalty for minor estimation errors. The final pitch estimate is derived through a weighted average of the output bins, enabling sub-bin precision.

Evaluation and Datasets

The authors evaluated CREPE using two primary datasets to ensure both baseline performance and real-world applicability:

  1. RWC-synth: Synthesized sinusoids providing a clean baseline.
  2. MDB-stem-synth: A realistic dataset of 230 monophonic stems from MedleyDB, re-synthesized to provide perfect ground truth labels while maintaining complex instrumental timbres.

Results and Discussion

CREPE demonstrates superior performance across multiple metrics compared to industry-standard algorithms like pYIN and SWIPE.

Accuracy and Precision

On the MDB-stem-synth dataset, CREPE achieved a Raw Pitch Accuracy (RPA) of 96.7%, outperforming pYIN (91.9%) and SWIPE (92.5%). Notably, its precision remains high even under strict evaluation criteria; at a 10-cent threshold, CREPE maintains a significant lead over traditional methods.

Noise Robustness

One of CREPE's most significant advantages is its stability in noisy environments. The model was tested against various noise types, including pub noise, white noise, and pink noise.

Performance of CREPE compared to pYIN and SWIPE across different noise types and signal-to-noise ratios (SNR). CREPE consistently maintains higher accuracy at low SNRs.

As shown in the figure above, CREPE consistently outperforms pYIN and SWIPE at low Signal-to-Noise Ratios (SNR < 10dB), highlighting the effectiveness of the learned convolutional features over hand-crafted DSP heuristics.

Conclusion and Future Work

CREPE represents a significant step forward in pitch estimation by demonstrating that a deep learning approach can surpass the performance ceiling of traditional DSP methods. While highly effective, the current model processes frames independently. Future work could incorporate temporal context through Recurrent Neural Networks (RNNs) to improve the smoothness of pitch trajectories. Additionally, expanding the training data to include more diverse instruments could further enhance the model's generalization capabilities.

Summary updated

BibTeX
@inproceedings{kim2018crepe,
  author = {Jong Wook Kim and Justin Salamon and Peter Li and Juan Pablo Bello},
  title = {{Crepe: A Convolutional Representation for Pitch Estimation}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2018},
  doi = {10.1109/icassp.2018.8461329},
}

Loading related content...

Figure