Abstract
Pitch estimation is a fundamental task in audio processing, traditionally dominated by hand-crafted digital signal processing (DSP) algorithms. This paper introduces CREPE (Convolutional Representation for Pitch Estimation), a deep learning-based pitch tracker that learns to estimate fundamental frequency ($f_0$) directly from raw audio waveforms. By utilizing a deep convolutional neural network, CREPE achieves state-of-the-art accuracy and demonstrates significantly improved robustness to noise compared to established methods like pYIN.
Introduction
Pitch estimation, or $f_0$ tracking, involves determining the fundamental frequency of a monophonic sound. Despite being a foundational task in audio engineering—used in speech analysis, music information retrieval, and more—it remains challenging due to complex timbres, rapid pitch fluctuations, and background noise. Traditional algorithms, such as the YIN algorithm or Autocorrelation Function (ACF), rely on human-designed heuristics that often fail in non-ideal acoustic environments. CREPE addresses these limitations by adopting a data-driven approach, learning pitch representations directly from raw audio data.
Methodology
The core of CREPE is a deep convolutional neural network (CNN) that operates directly on time-domain waveforms, bypassing the need for intermediate representations like spectrograms.
Architecture
The model consists of six convolutional layers followed by a fully connected output layer. It processes a 1024-sample window of audio at a 16 kHz sampling rate.
Output Representation and Training
Instead of a single scalar value, CREPE outputs a 360-dimensional vector. Each bin corresponds to a specific pitch value in cents, spanning six octaves. During training, the target output is a Gaussian-blurred curve centered on the ground truth pitch. This spatial smoothing allows the model to learn the relationships between adjacent pitch bins and reduces the penalty for minor estimation errors. The final pitch estimate is derived through a weighted average of the output bins, enabling sub-bin precision.
Evaluation and Datasets
The authors evaluated CREPE using two primary datasets to ensure both baseline performance and real-world applicability:
- RWC-synth: Synthesized sinusoids providing a clean baseline.
- MDB-stem-synth: A realistic dataset of 230 monophonic stems from MedleyDB, re-synthesized to provide perfect ground truth labels while maintaining complex instrumental timbres.
Results and Discussion
CREPE demonstrates superior performance across multiple metrics compared to industry-standard algorithms like pYIN and SWIPE.
Accuracy and Precision
On the MDB-stem-synth dataset, CREPE achieved a Raw Pitch Accuracy (RPA) of 96.7%, outperforming pYIN (91.9%) and SWIPE (92.5%). Notably, its precision remains high even under strict evaluation criteria; at a 10-cent threshold, CREPE maintains a significant lead over traditional methods.
Noise Robustness
One of CREPE's most significant advantages is its stability in noisy environments. The model was tested against various noise types, including pub noise, white noise, and pink noise.
As shown in the figure above, CREPE consistently outperforms pYIN and SWIPE at low Signal-to-Noise Ratios (SNR < 10dB), highlighting the effectiveness of the learned convolutional features over hand-crafted DSP heuristics.
Conclusion and Future Work
CREPE represents a significant step forward in pitch estimation by demonstrating that a deep learning approach can surpass the performance ceiling of traditional DSP methods. While highly effective, the current model processes frames independently. Future work could incorporate temporal context through Recurrent Neural Networks (RNNs) to improve the smoothness of pitch trajectories. Additionally, expanding the training data to include more diverse instruments could further enhance the model's generalization capabilities.
Summary updated