A Streamlined Encoder/decoder Architecture for Melody Extraction

Tsung-Han Hsieh, Li Su, Yi-Hsuan Yang

TL;DR

Researchers have developed a "streamlined" encoder-decoder neural network that simplifies melody extraction from complex polyphonic music. By using a clever "non-melody" detection branch and passing pooling indices through the network, the model achieves state-of-the-art accuracy with just 7 convolutional layers—a massive reduction compared to previous 45-layer models—while eliminating the need for manual thresholding.

What Problem Does This Paper Address?

Melody extraction is the process of identifying the fundamental frequency (F0) of the leading melody in a piece of music. This is notoriously difficult in polyphonic audio where multiple instruments and vocals overlap.

The authors identify two major pain points in current deep-learning approaches:

  1. Over-Complexity: Leading models are often "black boxes" with dozens of layers (e.g., 45 layers), making them computationally expensive and hard to interpret.
  2. The Binarization Headache: Most models output a "salience map" (a grid of probabilities). To get a final answer, researchers usually have to pick an arbitrary threshold to decide if a melody actually exists in a specific time frame. If the threshold is too low, you get "ghost" notes; too high, and you miss the melody entirely.

Methods

The authors propose a streamlined architecture based on an encoder-decoder structure, introducing two key technical innovations:

  • Pooling Indices for Localization: Inspired by SegNet (a computer vision model), the encoder records the location of the maximum values during "max-pooling" and passes these indices directly to the decoder's "un-pooling" layers. This helps the model precisely pinpoint where the melody is in the frequency domain without needing dozens of extra layers to "re-learn" that spatial information.
  • The "Non-Melody" Bottleneck Branch: Instead of just predicting the melody, the model has a dedicated branch at the bottleneck layer that estimates the probability that no melody is present. This "non-melody" estimate is concatenated to the final salience map as an extra row ($F+1$).
  • The Argmax Trick: Because of the extra "non-melody" row, the model can use a simple argmax function. If the highest value in a time frame is in the "non-melody" row, the system concludes there is no melody. This replaces the need for ad-hoc, empirical thresholding.

The model uses the Combined Frequency and Periodicity (CFP) representation as input, which combines spectrograms with generalized cepstrum data to provide a rich starting point for the network.

Datasets

The model was evaluated on several standard benchmarks:

  • MedleyDB: Used for both general melody extraction and vocal melody extraction.
  • ADC2004 & MIREX05: Classic datasets for vocal melody extraction.
  • MIR-1K: A dataset of 1,000 Chinese karaoke clips used for training the vocal models.

Key Insights and Ideas

  • Efficiency is Possible: You don't need 45 layers to achieve state-of-the-art results. By using architectural shortcuts like pooling indices, a 7-layer model can perform just as well, if not better.
  • Binarization as a Class: Treating "silence" or "no-melody" as just another category in a classification task (the $F+1$ bin) is much more robust than trying to find a "magic number" threshold after the model has finished running.
  • Structural Interpretation: Because the model is simpler, it is easier to see how each layer contributes to the final result, moving away from the "black box" nature of deeper architectures.

Key Results

  • Superior Accuracy: On the MedleyDB dataset, the proposed model achieved an Overall Accuracy (OA) of 79.7%, significantly outperforming the previous state-of-the-art (70.0%).
  • Reduced False Alarms: The built-in non-melody detector drastically reduced the Voicing False Alarm (VFA) rate. For example, on the ADC2004 dataset, it achieved a VFA of 19.2% compared to the 50.5% of the Deep Salience Model (DSM).
  • Speed and Simplicity: The model is lightweight enough to converge in just 20 minutes on a single consumer-grade GPU (GTX 1080ti).

Limitations and Future Work

While the model excels at deciding if a melody exists, the authors noted it is slightly weaker in "pitch-related" metrics like Raw Pitch Accuracy (RPA) on certain datasets. It occasionally struggles with high-frequency noise, leading to sporadic wrong predictions.

Future work could involve applying this "streamlined" philosophy to more complex tasks like Multi-Pitch Estimation (MPE), where multiple notes are active simultaneously, though the authors acknowledge that the simple argmax trick would need to be adapted for such scenarios.

Summary updated

Abstract

Melody extraction in polyphonic musical audio is important for music signal processing. In this paper, we propose a novel streamlined encoder/decoder network that is designed for the task. We make two technical contributions. First, drawing inspiration from a state-of-the-art model for semantic pixel-wise segmentation, we pass through the pooling indices between pooling and un-pooling layers to localize the melody in frequency. We can achieve result close to the state-of-the-art with much fewer convolutional layers and simpler convolution modules. Second, we propose a way to use the bottleneck layer of the network to estimate the existence of a melody line for each time frame, and make it possible to use a simple argmax function instead of ad-hoc thresholding to get the final estimation of the melody line. Our experiments on both vocal melody extraction and general melody extraction validate the effectiveness of the proposed model.
BibTeX
@inproceedings{hsieh2019streamlined,
  author = {Tsung-Han Hsieh and Li Su and Yi-Hsuan Yang},
  title = {{A Streamlined Encoder/decoder Architecture for Melody Extraction}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2019},
  url = {https://openreview.net/forum?id=7wXERhCPrb},
}

Loading related content...

Figure