Federating Song Form, Not Sound: Level-Split Training of Hierarchical Music Generators

Music generators can now produce convincing 10- to 30-second clips, but over several minutes they wander. They drift between keys, lose the beat, and never commit to a verse-chorus-bridge form. The data that could teach them richer forms, such as creators' private multitrack sessions and stems from regional or non-Western traditions, cannot be pooled because of copyright and licensing. Federated learning addresses exactly this situation. In federated learning, each data holder trains on its own machine and sends only model updates to a server that averages them. The catch is that federated training has so far been studied almost entirely on short audio classification, and there are good reasons to expect naive federation to hurt long-range structure first. Each creator holds only a handful of full-length songs. Edge devices can only afford short training windows. Averaging updates from creators with very different styles produces a muddy compromise.

This note proposes a specific way out. Use a hierarchical audio tokenizer that represents a song as a very slow "skeleton" stream (about 10 tokens per second) plus faster streams of acoustic detail. Then federate the two levels differently. The skeleton model is cheap enough to train on a whole three-minute song on a creator's laptop, about 1,800 tokens. It is shared and averaged across clients, because song form is the knowledge worth pooling. The acoustic levels, which encode each creator's timbres, rooms and mixing, stay local and personalized. The claim is that this split makes the long-horizon signal actually trainable in a federated setting. Without it, federation preserves local sound quality but erodes form.

Core Claim

Under non-identically distributed clients, end-to-end federated averaging of a music generator degrades structural coherence much more than it degrades short-window fidelity. Two causes drive this. Clients can only afford short crops of high-rate tokens, and structural gradients from different clients are averaged together with acoustic gradients that disagree strongly across recording conditions. Splitting aggregation by level of the token hierarchy closes most of this gap at a fraction of the communication cost. In that split, full-song context is used only at the cheap skeleton level, which is shared, while the acoustic levels are local. The split is also expected to beat the strongest non-federated alternative, a skeleton model trained centrally on public synthetic data, specifically on musical forms that public data underrepresents.

This combination is not covered by the closest work. PyramidCodec introduced the multi-scale residual tokenizer, with top-level tokens at 10 tokens per second and 70 tokens per second in total. It showed a GPT-2-style model generating structured music up to 3 minutes, but trained everything centrally on 5,000 hours of synthesized Lakh MIDI. We reuse its representation and supply what it does not study: the training regime, the data partition, and the per-level aggregation rule. FedAudio benchmarks federated keyword spotting, emotion and sound-event classification under non-IID splits and noise. Its findings that 10 dB noise cut keyword accuracy by 27.12% and that the adaptive server optimizer FedOPT is more stable than FedAvg are what motivate keeping acoustics out of the shared model. FedAudio contains no generation and no long sequences. FedHealth shares lower layers and personalizes upper layers for wearable activity recognition, reporting a 5.3-point gain over unpersonalized models. We borrow the share-versus-personalize idea but apply it along the time-scale axis of a token hierarchy rather than the depth axis of a classifier. The survey Advances and Open Challenges in Federated Foundation Models flags communication cost for large generators as an open problem. Sending only skeleton-model updates is our concrete answer for music.

Research Program

The system has three parts. The first is a shared tokenizer. A PyramidCodec-style codec is pretrained centrally on public synthesized Lakh MIDI audio and its top-level quantizer is frozen. Freezing it gives every client the same skeleton vocabulary, which is a precondition for averaging skeleton models meaningfully. The acoustic residual levels of the codec decoder may be fine-tuned locally.

The second part is a skeleton language model. It is trained federatedly with FedOPT on full-song skeleton sequences, under client-level non-IID partitions.

The third part is a set of per-client acoustic models. Each client trains its own model mapping skeleton tokens to residual tokens on short windows, conditioned on the skeleton, and never uploads it. As a middle ground, clients may upload a low-rank adapter on the skeleton model. Low-rank adapters are small trainable weight corrections; Instruct-MusicGen showed that structural editing of MusicGen needs only about 8% new parameters this way, trained centrally. That design lets us test whether genre-specific forms need personalization at the skeleton level too.

Before any generation results, we run a cheap diagnostic. During ordinary end-to-end federated training, we measure the cosine agreement between client updates separately for skeleton-level and acoustic-level parameters. The hypothesis behind the split is that skeleton updates agree across clients far more than acoustic updates. If they do not, because forms themselves differ sharply by client, the design shifts toward personalized skeleton adapters. The note reports that outcome either way.

Baselines

The naive reference point is local-only training. Each client fine-tunes the publicly pretrained hierarchical model on its own songs with no communication. This shows what federation buys a client with few full-length examples. We expect it to overfit form, either memorizing its handful of song structures or collapsing to one of them.

The first strong competitor is end-to-end FedAvg and FedOPT over all levels of the same hierarchical model, with clients training on the longest crop their memory allows at the full token rate. This isolates the key ingredient. It uses identical architecture, data partition, number of rounds, local steps and learning-rate sweep, so the only differences are per-level aggregation and full-song skeleton context. Beating it means higher structure-clarity and coherence scores at equal or better short-window FAD, with less upload per round.

The second competitor is federated low-rank fine-tuning of a flat music language model, the configuration that Instruct-MusicGen trained centrally. Here it would be trained with FedOPT and run with InfiniteAudio sliding-window inference, a training-free method that removes seams between blocks at constant memory, to reach three minutes. This is the strongest system someone could assemble today from the file's components. For fairness, its adapter is sized to match our per-round upload, and both systems receive the same tuning budget. The closest reported number is Instruct-MusicGen's FAD of 3.75 on Slakh, but that measures stem editing on short clips, not long-form generation. No reported result was found on Lacuna for federated music generation of any length. So this baseline's numbers will come from our own runs.

The third competitor is the centralized public skeleton. The skeleton model is trained only on public synthesized Lakh MIDI and frozen, with acoustic models personalized locally. This is PyramidCodec's centralized recipe combined with local personalization. It tests whether federating structure adds anything beyond what public data already provides.

An oracle sets the ceiling: the same hierarchical model trained on the pooled client data centrally.

Why not just…?

Why not just learn song form centrally from public MIDI, which is abundant? This is the most serious threat. Lakh MIDI provides over 170,000 files, and pop form is plausibly learnable from it. Our answer is empirical rather than rhetorical. We construct clients whose data comes from genres and forms deliberately held out of the public pretraining pool, and we use real multitracks from MoisesDB as a held-out real-audio domain. Mustango concedes that its Western-theory feature extractors fail on non-Western and microtonal structure, which suggests public-data priors carry biases. If the frozen public skeleton matches federated skeletons even on held-out forms, we will report that federation is only worth it for acoustics, and the contribution narrows to the local acoustic half.

Why not just fix drift at inference time with sliding-window sampling? InfiniteAudio smooths boundaries between consecutive windows but carries no plan for the whole song. BASS shows that even the best audio language model scores only 26.54% on its musical structure and reasoning tasks, confusing verses with choruses, so structure is not something current models recover implicitly. The second competitor above directly tests whether smoothing alone yields form. Our prediction is that it scores well on local naturalness and poorly on structure clarity.

Why not just condition on beat and chord tags from music-analysis tools? Mustango's beat and chord cross-attention reaches roughly 75% on its chord-matching test, but it was tuned for 10-second clips and relies on Western-theory extractors. Tags also say nothing about section-level form. Tag conditioning is complementary rather than competing, so we include a variant that feeds extracted key and tempo to the skeleton model. The question is whether the gains of the federated skeleton survive when the baselines get tags too.

Why not just run plain FedAvg, since adaptive optimizers already stabilize audio federation? FedAudio's stability results come from short classification tasks. The level-split ablation against end-to-end FedOPT is exactly the experiment that shows whether optimizer choice alone suffices for long-horizon generation.

Proposed Experiments

Clients are built from Slakh2100 (145 hours of stems rendered from Lakh MIDI) and synthesized Lakh MIDI audio. We partition them with Dirichlet splits, following FedAudio's protocol, over MIDI-derived style clusters and instrument categories, and add FedAudio-style additive noise to emulate varied home recording conditions. We sweep 10 to 100 clients and the Dirichlet concentration from mild to extreme non-IID. MoisesDB's 240 professionally mixed songs serve as small real-audio clients and as an out-of-domain test.

Long-form quality is scored with an aesthetic predictor trained on SongEval, which reports a Pearson correlation of 0.912 with professional musicians' averages, compared with 0.61–0.66 for existing metrics. We use its structure-clarity and overall-coherence dimensions. This is supplemented by a small blind listening study, because Aligning Generative Music AI with Human Preferences cautions that objective distances miss macro-structure. We also add simple measurable structure statistics: key stability over time using the key detector in MIRFLEX from SonicVerse, tempo stability, and self-similarity repetition of sections. We avoid audio language models as judges, given the BASS results. Short-window fidelity is measured by FAD on 10-second crops, and each client's personalization by FAD against its own held-out songs. Communication is measured as bytes uploaded per round.

Three results decide the question. First, end-to-end federation must show a larger relative drop from the oracle in structure clarity than in short-window FAD; without this the motivating premise fails. Second, the level-split method must recover most of that structural gap at equal or lower upload than end-to-end federation and the federated flat-adapter competitor. Third, it must beat the frozen public skeleton on clients whose forms are held out of public data. If it does not, federating structure is unnecessary and we will say so. The update-agreement diagnostic, read alongside the non-IID sweep, will show where the boundary lies between shareable form and form that must be personalized.