A Streamlined Encoder/Decoder Architecture for Melody Extraction

Tsung-Han Hsieh, Li Su, Yi-Hsuan Yang

Abstract

Melody extraction in polyphonic musical audio is important for music signal processing. In this paper, we propose a novel streamlined encoder/decoder network that is designed for the task. We make two technical contributions. First, drawing inspiration from a state-of-the-art model for semantic pixel-wise segmentation, we pass through the pooling indices between pooling and un-pooling layers to localize the melody in frequency. We can achieve result close to the state-of-the-art with much fewer convolutional layers and simpler convolution modules. Second, we propose a way to use the bottleneck layer of the network to estimate the existence of a melody line for each time frame, and make it possible to use a simple argmax function instead of ad-hoc thresholding to get the final estimation of the melody line. Our experiments on both vocal melody extraction and general melody extraction validate the effectiveness of the proposed model.

Introduction

Melody extraction—the process of identifying the fundamental frequency (F0) of the leading melody in polyphonic audio—is a significant challenge in music information retrieval. Current deep-learning approaches often suffer from two main drawbacks: over-complexity and the "binarization headache." Leading models frequently employ dozens of layers (e.g., 45 layers), making them computationally expensive and difficult to interpret. Furthermore, these models typically output a salience map that requires manual, ad-hoc thresholding to determine if a melody is present in a given frame. Incorrect thresholds lead to either "ghost" notes or missed detections.

Methodology

The proposed architecture addresses these issues through a streamlined encoder-decoder structure that achieves high performance with only seven convolutional layers.

Localization via Pooling Indices

Inspired by SegNet, the model records the spatial locations of maximum values during the encoder's max-pooling stages. These "pooling indices" are passed directly to the corresponding un-pooling layers in the decoder. This mechanism allows the network to precisely localize melody components in the frequency domain without the need for the deep, redundant layers typically used to recover spatial information.

The proposed streamlined encoder-decoder architecture featuring the non-melody estimate branch and pooling index connections.

The Non-Melody Bottleneck Branch and Argmax Trick

A critical innovation is the introduction of a dedicated branch at the bottleneck layer designed to estimate the probability of "non-melody" frames. This estimate is concatenated to the final salience map as an additional row ($F+1$). By including the non-melody estimate directly in the output map, the model transforms the detection problem into a simple classification task. Instead of empirical thresholding, a standard argmax function is applied across the $F+1$ bins. If the highest probability falls in the $F+1$ bin, the system concludes no melody is present. This approach significantly simplifies the post-processing pipeline and improves robustness.

Experiments and Results

The model was trained using the Combined Frequency and Periodicity (CFP) representation and evaluated on standard benchmarks including MedleyDB, ADC2004, and MIREX05. The model is lightweight enough to converge in just 20 minutes on a single consumer-grade GPU (GTX 1080ti).

Performance Metrics

The proposed model demonstrates superior accuracy and efficiency. On the MedleyDB dataset, it achieved an Overall Accuracy (OA) of 79.7%, a substantial improvement over the previous state-of-the-art of 70.0%. Notably, the built-in non-melody detector significantly reduced Voicing False Alarms (VFA).

Performance comparison on the ADC2004 dataset showing the proposed model's high accuracy and low false alarm rates.

As shown in the table above, the proposed model achieves a VFA of 19.2% on the ADC2004 dataset, drastically outperforming the Deep Salience Model (DSM), which recorded a VFA of 50.5%. While the model is slightly less dominant in Raw Pitch Accuracy (RPA) on certain subsets, its overall robustness and simplicity are clear advantages.

Conclusion

This research demonstrates that melody extraction does not require massive, 45-layer "black box" architectures. By utilizing architectural shortcuts like pooling indices and treating non-melody detection as an integrated classification task, the proposed 7-layer model provides a faster, more interpretable, and more accurate solution. Future work will explore adapting the argmax classification strategy for Multi-Pitch Estimation (MPE) tasks.

Summary updated

Abstract

Melody extraction in polyphonic musical audio is important for music signal processing. In this paper, we propose a novel streamlined encoder/decoder network that is designed for the task. We make two technical contributions. First, drawing inspiration from a state-of-the-art model for semantic pixel-wise segmentation, we pass through the pooling indices between pooling and un-pooling layers to localize the melody in frequency. We can achieve result close to the state-of-the-art with much fewer convolutional layers and simpler convolution modules. Second, we propose a way to use the bottleneck layer of the network to estimate the existence of a melody line for each time frame, and make it possible to use a simple argmax function instead of ad-hoc thresholding to get the final estimation of the melody line. Our experiments on both vocal melody extraction and general melody extraction validate the effectiveness of the proposed model.
BibTeX
@inproceedings{hsieh2019streamlined,
  author = {Tsung-Han Hsieh and Li Su and Yi-Hsuan Yang},
  title = {{A Streamlined Encoder/Decoder Architecture for Melody Extraction}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2019},
  url = {https://openreview.net/forum?id=7wXERhCPrb},
}
Figure