Abstract
Modeling the human voice within complex musical arrangements remains a fundamental challenge in Music Information Retrieval (MIR). This paper introduces Mel-RoFormer, a novel Transformer-based architecture designed to excel in two critical tasks: vocal separation and vocal melody transcription. By integrating a biologically-inspired Mel-band frequency projection with interleaved time-frequency Transformers and Rotary Position Embeddings (RoPE), the model captures complex spectral relationships more effectively than traditional CNN-based approaches. Furthermore, the study demonstrates that vocal separation serves as a powerful foundation task for transcription, leading to state-of-the-art results on benchmarks like MUSDB18HQ and MIR-ST500.
Introduction
Music audio signals are notoriously difficult to model because they contain a dense overlap of melodies, harmonics, and varying timbres. In the field of MIR, two "holy grail" tasks are vocal separation—isolating the singer from the instrumental background—and vocal melody transcription—converting that singing into digital note representations.
Traditional approaches have largely relied on Convolutional Neural Networks (CNNs) to treat audio spectrograms as 2D images. However, CNNs often struggle to capture long-range relationships between spectral patterns that are far apart in frequency. Additionally, vocal transcription suffers from a chronic "data scarcity" problem, as high-quality, human-annotated transcription data is significantly harder to produce than raw audio for separation. Mel-RoFormer addresses these issues through architectural innovation and a strategic approach to transfer learning.
Methodology
Mel-band Projection
Mel-RoFormer builds upon the Band-Split RoFormer (BS-RoFormer) but introduces a critical change in how frequency information is processed. Instead of splitting the frequency spectrum into arbitrary, non-overlapping linear chunks, the model utilizes the Mel-scale. This scale mimics human hearing by providing higher resolution at lower frequencies where vocal nuances are most prominent. These overlapping bands are processed by Multi-Layer Perceptrons (MLPs) to create a sequence of "band-wise" features that serve as the input for the Transformer blocks.
Interleaved RoPE Transformers
The core of the architecture consists of a stack of Transformer blocks that alternate their focus. One sub-block processes data across the time dimension, while the subsequent sub-block processes it across the frequency (band) dimension. To maintain spatial and temporal awareness, the model utilizes Rotary Position Embedding (RoPE). This allows the model to remain invariant to the rearrangement of audio segments while keeping track of the relative positions of features within the sequence.
The Foundation Model Strategy
To solve the data scarcity problem in transcription, the authors adopted a two-stage training strategy. They first trained a massive vocal separation model on large-scale datasets. Once the model learned to perfectly isolate a voice from a noisy background, the core weights were "frozen" and fine-tuned for transcription. The hypothesis is that a model capable of high-fidelity separation has already mastered the essential features required to identify pitch and timing.
Experimental Results
The researchers evaluated Mel-RoFormer across several industry-standard benchmarks, including MUSDB18HQ and MoisesDB for separation, and MIR-ST500 and POP909 for transcription.
Vocal Separation Performance
On the MUSDB18HQ test set, Mel-RoFormer achieved a Signal-to-Distortion Ratio (SDR) of 13.29 dB. This result established a new state-of-the-art (SOTA), outperforming previous leaders such as BSRNN and the original BS-RoFormer. Even the "small" variant of the model (9.1M parameters) achieved over 11 dB SDR, demonstrating high efficiency for resource-constrained applications.
Transcription Breakthroughs
The impact of the foundation model strategy was most evident in the transcription results. On the MIR-ST500 dataset, the large Mel-RoFormer achieved a 7.5 percentage point increase in the strict COnPOff metric (Correct Onset, Pitch, and Offset) compared to the previous SOTA, SpecTNT. The model showed particular strength in identifying "offsets"—the exact moment a singer stops a note—which is traditionally one of the most difficult aspects of digital transcription.
Discussion and Conclusion
The success of Mel-RoFormer highlights two key insights: bio-inspired frequency division (Mel-scaling) significantly improves a model's ability to generalize to unseen music, and vocal separation is an excellent pre-text task for broader MIR applications. Fine-tuning a separation model for transcription was found to converge faster (under 15k steps) and perform better than training a transcription model from scratch for over 50k steps.
While the model is highly effective, performance remains somewhat limited by the inherent "noisiness" and quality of ground-truth labels in existing transcription datasets. Future work will explore using Mel-RoFormer as a backbone for other tasks, such as chord recognition and multi-instrument source separation, potentially serving as a universal foundation for music understanding.
Summary updated