MusicRWKV: A Multimodal RWKV Framework for Music Representation Learning

Terry TanarXiv 2026 · arXiv

TL;DR

MusicRWKV is a new multimodal framework for music representation learning that replaces the standard Transformer backbone with the more efficient RWKV architecture. By combining linear-time inference with a novel invertible compression module and FiLM-based modality fusion, it achieves competitive performance on diverse music understanding tasks while significantly reducing the computational bottlenecks associated with long-duration audio.

What Problem Does This Paper Address?

In the world of Music Information Retrieval (MIR), "foundation models" like MERT and MuQ have set the bar for learning transferable representations from raw audio. However, these models almost exclusively rely on Transformer architectures.

The problem? Transformers suffer from quadratic attention complexity ($O(N^2)$). As music recordings often span several minutes, high-resolution audio tokenizers produce extremely long sequences. This leads to massive computational overhead and a memory bottleneck caused by Key-Value (KV) caching during inference. The authors of MusicRWKV aim to break this "Transformer dependency" by exploring a more scalable architectural alternative that can handle long-range dependencies without the $O(N^2)$ cost.

Methods

The core of the framework is the RWKV (Receptance Weighted Key Value) architecture. RWKV is a hybrid that offers the best of both worlds: it can be trained in parallel like a Transformer but performs inference with the linear complexity and constant memory footprint of a Recurrent Neural Network (RNN).

Figure 1: Illustration of MusicRWKV Framework

Key technical innovations include:

  • Multimodal Input: The model jointly processes three streams: Waveform tokens (via Jukebox), Spectrogram tokens (via MuQ Mel-RVQ), and Metadata (genre, artist, lyrics).
  • Invertible Audio Compression: To further boost efficiency, the authors introduced a module that groups adjacent audio tokens into fixed-length segments, projecting them into a latent space. This reduces the sequence length handled by the encoders while maintaining the ability to reconstruct the original signal.
  • FiLM-based Conditioning: Instead of using expensive cross-attention for multimodal fusion, the model uses Feature-wise Linear Modulation (FiLM). This applies learnable affine transformations to modulate features based on the modality, facilitating interaction at a much lower computational cost.
  • Soft ROSA: The authors experimented with an optional "Soft ROSA" mechanism—a differentiable retrieval module that allows the network to selectively "pull" information from historical candidate representations, acting as a router-free expert selection system.

Datasets

  • Pretraining: The model was pretrained on a genre-stratified corpus of approximately 25,000 full-length music recordings collected from the internet.
  • Downstream Evaluation: To test generalizability, the authors used a wide array of standard MIR benchmarks:
  • MagnaTagATune (MTT): Music tagging.
  • GTZAN: Genre classification.
  • GiantSteps Key: Musical key estimation.
  • EMO: Emotion recognition.
  • NSynth: Instrument and pitch classification.
  • VocalSet: Singing voice analysis (singer ID and vocal technique).

Key Insights and Ideas

  • RWKV is a Viable Transformer Alternative: The results prove that the quadratic complexity of Transformers isn't a prerequisite for high-quality music SSL. MusicRWKV matches or exceeds Transformer-based SOTA models on several tasks.
  • The "Selective Retrieval" Shift: One of the most interesting findings came from "layer-wise telemetry" of the Soft ROSA module. The authors found that early layers maintain "distributed" retrieval (looking at many historical candidates), while deeper layers become highly "selective," focusing on a small subset of relevant information. This suggests a hierarchical refinement of musical concepts.
  • Capacity Mismatch: The authors noted that directly fine-tuning the large pretrained model on small downstream datasets often led to overfitting. They found better success by using the MusicRWKV architecture but training smaller, task-specific versions from scratch.

Key Results

MusicRWKV achieved competitive results across the board, with particularly standout performance in specific areas compared to Transformer baselines like MERT and MuQ:

  • Music Tagging (MTT): Achieved an Average Precision (AP) of 46.1, significantly higher than MERT (37.9) and MuQ (38.5).
  • Key Estimation (GiantSteps): Reached 70.3% refined accuracy, outperforming MERT (64.1%) and AudioMAE++ (61.7%).
  • Vocal Technique (VocalSet): Hit 89.1% accuracy, a substantial jump over MERT's 76.9%.
  • Ablation: Incorporating the Soft ROSA module improved pretraining validation accuracy from 93.3% to 94.8%, confirming the value of the differentiable retrieval mechanism.

Limitations and Future Work

While promising, the authors acknowledge several hurdles:

  • Soft ROSA Overhead: While it improves accuracy, Soft ROSA introduces its own computational and memory costs, which could limit scalability for extremely long contexts.
  • Scaling Laws: This study focused on a single large configuration (~883M parameters). Future work is needed to see how MusicRWKV scales—does performance continue to improve linearly with model size?
  • Modality Selection: The results suggest that a "one-size-fits-all" multimodal representation might not be optimal. Some tasks benefit more from specific modalities (e.g., pitch classification relies heavily on CQT). Future iterations might explore adaptive modality selection.

Summary updated

Abstract

Recent music foundation models have achieved remarkable progress in music representation learning, yet most existing approaches rely on Transformer architectures whose quadratic attention complexity introduces substantial computational and memory overhead when modeling long-duration music recordings. In this work, we propose MusicRWKV, a multimodal music representation learning framework built upon the RWKV architecture. MusicRWKV jointly models waveform, spectrogram, and metadata information within a unified self-supervised learning framework. To improve efficiency for long musical sequences, we introduce an invertible compression module that reduces sequence length while preserving reconstruction capability. A FiLM-based conditioning strategy is employed to facilitate modality-aware representation learning without introducing expensive cross-attention operations. We further investigate the integration of Soft ROSA as a differentiable retrieval mechanism for neural-symbolic interaction. Experimental evaluation across a diverse collection of Music Information Retrieval (MIR) benchmarks demonstrates that the proposed RWKV-based architecture can be effectively adapted to heterogeneous music understanding tasks. A controlled pretraining ablation shows that Soft ROSA improves validation performance, while layer-wise telemetry analysis reveals a progressive transition from distributed retrieval to selective retrieval in deeper network layers. These results suggest that RWKV provides a promising architectural alternative to Transformer-based models for multimodal music understanding.
BibTeX
@misc{tan2026musicrwkv,
  author = {Terry Tan},
  title = {{MusicRWKV: A Multimodal RWKV Framework for Music Representation Learning}},
  year = {2026},
  eprint = {2611.16718},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.16718},
}
Figure