Mel-RoFormer for Vocal Separation and Vocal Melody Transcription

Ju-Chiang Wang, Wei-Tsung Lu, Jitong Chen

Abstract

Modeling the human voice within complex musical arrangements remains a fundamental challenge in Music Information Retrieval (MIR). This paper introduces Mel-RoFormer, a novel Transformer-based architecture designed to excel in two critical tasks: vocal separation and vocal melody transcription. By integrating a biologically-inspired Mel-band frequency projection with interleaved time-frequency Transformers and Rotary Position Embeddings (RoPE), the model captures complex spectral relationships more effectively than traditional CNN-based approaches. Furthermore, the study demonstrates that vocal separation serves as a powerful foundation task for transcription, leading to state-of-the-art results on benchmarks like MUSDB18HQ and MIR-ST500.

Introduction

Music audio signals are notoriously difficult to model because they contain a dense overlap of melodies, harmonics, and varying timbres. In the field of MIR, two "holy grail" tasks are vocal separation—isolating the singer from the instrumental background—and vocal melody transcription—converting that singing into digital note representations.

Traditional approaches have largely relied on Convolutional Neural Networks (CNNs) to treat audio spectrograms as 2D images. However, CNNs often struggle to capture long-range relationships between spectral patterns that are far apart in frequency. Additionally, vocal transcription suffers from a chronic "data scarcity" problem, as high-quality, human-annotated transcription data is significantly harder to produce than raw audio for separation. Mel-RoFormer addresses these issues through architectural innovation and a strategic approach to transfer learning.

Methodology

Mel-band Projection

Mel-RoFormer builds upon the Band-Split RoFormer (BS-RoFormer) but introduces a critical change in how frequency information is processed. Instead of splitting the frequency spectrum into arbitrary, non-overlapping linear chunks, the model utilizes the Mel-scale. This scale mimics human hearing by providing higher resolution at lower frequencies where vocal nuances are most prominent. These overlapping bands are processed by Multi-Layer Perceptrons (MLPs) to create a sequence of "band-wise" features that serve as the input for the Transformer blocks.

Interleaved RoPE Transformers

The core of the architecture consists of a stack of Transformer blocks that alternate their focus. One sub-block processes data across the time dimension, while the subsequent sub-block processes it across the frequency (band) dimension. To maintain spatial and temporal awareness, the model utilizes Rotary Position Embedding (RoPE). This allows the model to remain invariant to the rearrangement of audio segments while keeping track of the relative positions of features within the sequence.

The Foundation Model Strategy

To solve the data scarcity problem in transcription, the authors adopted a two-stage training strategy. They first trained a massive vocal separation model on large-scale datasets. Once the model learned to perfectly isolate a voice from a noisy background, the core weights were "frozen" and fine-tuned for transcription. The hypothesis is that a model capable of high-fidelity separation has already mastered the essential features required to identify pitch and timing.

Experimental Results

The researchers evaluated Mel-RoFormer across several industry-standard benchmarks, including MUSDB18HQ and MoisesDB for separation, and MIR-ST500 and POP909 for transcription.

Vocal Separation Performance

On the MUSDB18HQ test set, Mel-RoFormer achieved a Signal-to-Distortion Ratio (SDR) of 13.29 dB. This result established a new state-of-the-art (SOTA), outperforming previous leaders such as BSRNN and the original BS-RoFormer. Even the "small" variant of the model (9.1M parameters) achieved over 11 dB SDR, demonstrating high efficiency for resource-constrained applications.

Transcription Breakthroughs

The impact of the foundation model strategy was most evident in the transcription results. On the MIR-ST500 dataset, the large Mel-RoFormer achieved a 7.5 percentage point increase in the strict COnPOff metric (Correct Onset, Pitch, and Offset) compared to the previous SOTA, SpecTNT. The model showed particular strength in identifying "offsets"—the exact moment a singer stops a note—which is traditionally one of the most difficult aspects of digital transcription.

Discussion and Conclusion

The success of Mel-RoFormer highlights two key insights: bio-inspired frequency division (Mel-scaling) significantly improves a model's ability to generalize to unseen music, and vocal separation is an excellent pre-text task for broader MIR applications. Fine-tuning a separation model for transcription was found to converge faster (under 15k steps) and perform better than training a transcription model from scratch for over 50k steps.

While the model is highly effective, performance remains somewhat limited by the inherent "noisiness" and quality of ground-truth labels in existing transcription datasets. Future work will explore using Mel-RoFormer as a backbone for other tasks, such as chord recognition and multi-instrument source separation, potentially serving as a universal foundation for music understanding.

Summary updated

Abstract

Developing a versatile deep neural network to model music audio is crucial in MIR. This task is challenging due to the intricate spectral variations inherent in music signals, which convey melody, harmonics, and timbres of diverse instruments. In this paper, we introduce Mel-RoFormer, a spectrogram-based model featuring two key designs: a novel Mel-band Projection module at the front-end to enhance the model's capability to capture informative features across multiple frequency bands, and interleaved RoPE Transformers to explicitly model the frequency and time dimensions as two separate sequences. We apply Mel-RoFormer to tackle two essential MIR tasks: vocal separation and vocal melody transcription, aimed at isolating singing voices from audio mixtures and transcribing their lead melodies, respectively. Despite their shared focus on singing signals, these tasks possess distinct optimization objectives. Instead of training a unified model, we adopt a two-step approach. Initially, we train a vocal separation model, which subsequently serves as a foundation model for fine-tuning for vocal melody transcription. Through extensive experiments conducted on benchmark datasets, we showcase that our models achieve state-of-the-art performance in both vocal separation and melody transcription tasks, underscoring the efficacy and versatility of Mel-RoFormer in modeling complex music audio signals.
BibTeX
@inproceedings{wang2024mel,
  author = {Ju-Chiang Wang and Wei-Tsung Lu and Jitong Chen},
  title = {{Mel-RoFormer for Vocal Separation and Vocal Melody Transcription}},
  booktitle = {ISMIR},
  year = {2024},
  url = {https://openreview.net/forum?id=vybLHmxXeN},
}
Figure