Pre-Training Music Encoders on Features That Can Legally Be Shared

The strongest music understanding models are trained on audio nobody else can download. MERT's best model learned from 160,000 hours of non-public music, and the only openly reproducible version was trained on 910 hours of Music4All (MERT). At the same time, the field already shares one large commercial music corpus in a legal form. The Melon Playlist Dataset distributes 48-band mel-spectrograms for 649,091 commercial tracks. A mel-spectrogram is a coarse picture of how energy spreads across frequencies over time, and it cannot easily be turned back into listenable audio (Melon Playlist Dataset). That data has so far been used only for small task-specific networks. We propose to treat it the way domain-specific language modeling treats a specialized corpus. We will pre-train a music encoder from scratch only on these shareable features, then measure precisely what the restriction to 48 pre-computed bands costs and what commercial-grade data buys.

The core claim is testable. A self-supervised encoder trained only on Melon's 48-band spectrograms will match or beat the openly reproducible waveform model, MERT-95M-public, on the MARBLE music benchmark at matched model size and compute. Any remaining deficit will sit mainly in pitch- and score-level tasks, where 48 bands are too coarse, and not in genre, mood, or tagging tasks. The contribution is the model together with a controlled decomposition of the "accessibility tax" into two parts. The first is the cost of the feature format (48-band mel instead of waveform). The second is the gain from the data source (commercial production instead of Creative Commons audio). Anyone can rerun both, because every training input is redistributable.

Why this gap is real, and what constrains the design

Three lines of prior work motivate the proposal. The first argues about fixed features. freefield1010 contends that pre-computed features lock researchers into old signal-processing choices and block end-to-end learning. MelHuBERT answers this for speech. It replaced HuBERT's waveform front-end with 40-dimensional log-mel input and a cross-entropy loss over k-means cluster IDs. The result was 11.9% versus 12.6% word error on LibriSpeech 360h and 19.2% versus 32.0% phone error at 100h, with 31.2% less pre-training time. MelHuBERT was tested only on speech, however. It also got worse at speaker identification (33.7% error versus HuBERT's 29.7%), which warns that fine timbre detail can be lost in fixed mel bands. We take this as both method and constraint. We reuse its recipe, and we expect timbre-sensitive music tasks to be where problems appear.

The second line shows that audio can be modeled like language directly on spectrograms. BEATs predicts discrete labels for masked spectrogram patches and refines its labeler iteratively (50.6% mAP on AudioSet-2M). NAPE predicts the next spectrogram patch's embedding with a causal Transformer (50.2 mAP on AudioSet-2M). Both use general audio at their own mel resolution and never at 48 bands. Neither targets music.

The third line supplies the domain-pretraining logic. MonoCoder trained a roughly 0.9B-parameter model only on high-performance-computing C/C++ code and outperformed larger general models on that code. BioBART continued pre-training on PubMed and raised MeQSum Rouge-L from 51.05 to 53.15. The general lesson is that a narrower in-domain corpus can beat scale, and that the pre-training objective should respect how the domain is structured. BioBART found that removing sentence shuffling mattered for exactly this reason.

The closest music work constrains our design choices. MERT's targets are EnCodec tokens and Constant-Q Transform features. Both require the waveform, so neither can be computed from Melon, and this forces a different objective. CultureMERT shows that the choice of data domain matters. Continual pre-training on 650 hours of non-Western music gave a 4.9% boost on non-Western tagging. It also names the frozen Western EnCodec codebook as a bottleneck. Our targets are learned from the training features themselves, which sidesteps that particular bottleneck.

Research Program

We build one encoder family: a 95M-parameter Transformer, matched to MERT-95M-public. It takes 48-band log-mel frames computed with Melon's Essentia pipeline. Melon clips run 20–50 seconds, so the corpus holds roughly 3,600–9,000 hours by the dataset's own figures. That is several times the 910 hours behind MERT-95M-public.

The primary objective follows MelHuBERT's masked prediction of k-means cluster IDs over mel frames, with one BEATs-style re-clustering round on the first-stage model's hidden states. We stop at one round because BEATs' repeated rounds are expensive. As an objective ablation, we also train a NAPE-style causal next-patch model on identical data. This tells us whether the result depends on the specific pre-training loss.

Melon also contains 148,826 human-curated playlists, and no waveform corpus offers this kind of signal. We use it as an optional second objective: a contrastive loss that pulls together clip embeddings from tracks that appear in the same playlist. It is reported strictly as an ablation, because it could explain gains on Melon's own tasks without saying anything about the features.

All downstream evaluation audio is passed through the identical 48-band Essentia pipeline. The model therefore never sees any other input format, and every score reflects what a feature-only model can do.

Baselines

The naive reference point is a randomly initialized version of the same encoder, trained only on each task's labels over 48-band inputs. We also include the supervised mel-CNN used in the Melon paper. On Melon's cold-start task, that CNN maps audio to collaborative-filtering vectors (the track representations learned from listening patterns) with a reported MSE of 0.0098 (Melon Playlist Dataset). Pre-training has to beat this number on Melon's own task or it has added nothing.

The strongest competitor on the actual question is MERT-95M-public (MERT). It is the only openly reproducible music encoder of the same size, and it has full waveform access. We evaluate it on MARBLE's tasks with MARBLE's probing protocol and the same probe-tuning budget per task as our model. No reported result found on Lacuna for MERT-95M-public's MARBLE scores, so we will run it ourselves. Beating it means a higher MARBLE average over tasks that do not need source separation, with no individual task category worse by more than its seed variance.

We also report MERT-v1-330M as an upper reference, trained on 160,000 proprietary hours. MARBLE reports 94.4% NSynth pitch-classification accuracy for it (MARBLE). We do not expect to match it on pitch, and we say so in advance.

The central ablation isolates the data source from the feature format, using pre-training corpora matched in hours and compute. One arm uses open Creative Commons music (MTG-Jamendo and FMA) as raw waveforms, with MelHuBERT's recipe swapped for HuBERT's waveform front-end. The second arm uses the same open audio rendered to 48-band mel. The third arm uses Melon's 48-band mel. The gap between the first two arms is the feature-format tax. The gap between the second and third arms is the value of commercial-grade data in shareable form.

Why not just…?

Why not just fine-tune MERT-95M-public on each task? This is the default. It is how the comparison above is run, since probing and fine-tuning MERT is exactly the baseline. Evidence from language suggests domain pre-training dominates the choice of fine-tuning method (PEFT for Python Code Generation). In music, though, this is the claim under test rather than a given. If MERT-95M-public wins across MARBLE, the proposal's main claim fails, and we will report that.

Why not just pre-train on Creative Commons waveforms, which are already legal? This is the most serious threat. Open catalogs are shareable and keep the full waveform. The file notes that open-licensed audio underrepresents commercial production standards and global traditions (CultureMERT), but that note is a claim, not a measurement. Our three-arm ablation settles the matter directly. If the open-waveform arm matches the Melon-mel arm, the right conclusion is "use open waveforms," and Melon adds nothing.

Why not just add Melon features to a waveform model's training data, or distill MERT-330M into a mel student? A waveform model cannot take mel input without a new front-end, and that amounts to our model. Distillation needs the teacher to process raw audio for every training clip, and Melon never provides raw audio. Any distilled student would have to be trained on some other audio, which defeats the point of reproducibility.

Proposed Experiments

Pre-training runs use equal step counts and batch sizes across all arms, with three seeds for the probes. Evaluation covers MARBLE's acoustic, performance, score, and high-level tasks, all rendered at 48 bands. Source separation is excluded, because its targets need waveforms. Melon-specific evaluation covers tagging over Melon's 30,652 tags and the cold-start mapping, both scored against the 0.0098 MSE baseline.

As a timbre check, echoing MelHuBERT's speaker-identification drop, we evaluate singer-trait classification on STraDa wherever its excerpts can be put through our pipeline. STraDa reports 89.8% singer-sex accuracy with a supervised model, which gives a reference level.

The first decisive check is whether Melon-mel beats MERT-95M-public on MARBLE's average. The second is whether the feature-format tax is concentrated in pitch and score tasks such as NSynth pitch, as the resolution argument predicts. If it is spread uniformly instead, the problem is not frequency resolution, and a finer shareable format would not help. The third decisive check is a scaling curve: Melon-mel trained on 10%, 30%, and 100% of the corpus. If performance is still rising at full size, releasing more feature-only commercial data is worth pursuing. Together these experiments tell the community whether shared features can replace shared audio for pre-training, and where they cannot.