Songified Guidance for Federated Lyrics Transcription

Lyrics transcription, which turns sung audio into the words being sung, works far worse than speech recognition. Singers hold vowels for whole notes, range widely in pitch, add vibrato, and usually sing over music. The data that would close this gap already exists in large quantities: the vocal takes people record in karaoke and practice apps on their own phones. That data is also a voiceprint, and pooling it on one server creates the kind of high-value target that privacy law and good engineering both warn against. Federated learning offers a way out. It is a training scheme where each device updates the model on its own audio and sends only weights to a server for merging. The trouble is that the private singing on devices is highly varied across people, so plain weight averaging tends to drift and stall.

This note proposes a specific way for the server to merge these updates well without ever hearing anyone sing. Devices send only coarse, noise-protected summaries of their singing: a histogram of pitch range and a histogram of how long vowels are held. The server uses these summaries to turn public read speech into synthetic "singing" that matches the private population's statistics. It then uses this synthetic set to choose how to weight and combine the device models. The claim is that guidance data matched to the singing domain is what makes server-side merging help rather than hurt when the private data sits on the far side of the speech-to-singing gap. Raw speech guidance should mislead the merge, and statistically matched songified speech should not.

Core Claim

Two existing results meet here. Guided Model Merging for Hybrid Data Learning (Feddle) keeps an "atlas" of recent client models on the server. It searches for merging coefficients, which may be negative, on a small central dataset, and it adds a fallback term that anchors the result to ordinary federated averaging. On CIFAR-100 under heavy communication delay it reports about 70.5% accuracy against 52.6% for FedAvg, and it holds up even when the central data is out of domain. That evidence comes entirely from images, where the central and client inputs share one space. PDAugment turns LibriSpeech into singing-like audio by moving pitch onto MIDI notes and stretching only vowels. That centralized augmentation brings DSing30 word error rate to 9.8%, but polyphonic Dali stays at 54.0%, which shows that synthetic singing alone leaves a large gap to real song.

Our hypothesis is that in audio, the domain of the guidance set decides whether guided merging works at all. If the server scores candidate merges on read speech, it will prefer coefficients that pull the model back toward speech. That is exactly the wrong direction for singing clients, and Feddle's out-of-domain robustness should not carry over across a gap this large. If the guidance set is songified speech whose pitch and duration statistics are fitted to privately released client histograms, the server's ranking of candidate merges should track the true client singing error closely enough for guided merging to beat federated averaging. What is new is the use of privatized acoustic statistics to build a public, domain-matched guidance set. We also test directly whether guidance quality, not merging itself, is the deciding variable.

Concretely, with atlas models $\theta_1,\dots,\theta_M$, FedAvg model $\bar\theta$, and guidance set $G_h$ built from public speech under client statistics $h$, the server solves

$$\min_{c}\; \mathcal{L}_{G_h}\Big(\textstyle\sum_{m} c_m \theta_m\Big) + \lambda \Big\|\textstyle\sum_m c_m\theta_m - \bar\theta\Big\|^2,$$

where $c_m$ are unconstrained merging coefficients, $\mathcal{L}_{G_h}$ is the transcription loss on the guidance set, and $\lambda$ sets how strongly the merge stays near plain averaging. This follows Feddle's formulation. Our contribution is the construction of $G_h$.

Research Program

The model is a Conformer-Transformer transcriber initialized from PDAugment-style centralized pretraining on songified LibriSpeech. This starting point is public and contains no private data. Clients are simulated by partitioning real singing corpora so that the data is non-identically distributed. DSing30 supplies monophonic karaoke clients. Dali supplies polyphonic clients with backing tracks. GTSinger supplies 20 professional singers in 9 languages with labelled techniques such as vibrato, falsetto and glissando, partitioned by singer so that each client has a distinct style. Partitions follow performer metadata where it is available and fall back to song otherwise.

Each round, every client computes two histograms on its own audio: fundamental-frequency range per phrase, and vowel-to-consonant duration ratio. Vowel boundaries come from a forced aligner run locally. The client adds Laplace noise to these histograms, which gives a formal $\epsilon$-differential-privacy guarantee on what the summaries reveal. Here $\epsilon$ is the standard bound on how much any single recording can change the released output. The server aggregates the summaries and resamples the PDAugment pitch and duration targets so that the songified guidance set reproduces them. It then runs the coefficient search above. As a stress test, client weight updates are additionally trained with DP-SGD, meaning per-example gradient clipping plus Gaussian noise as used in FL-VetTrans. FL-VetTrans is the only federated audio system in the file, and it handled only classification of 346 poultry recordings. We test whether guided merging recovers some of the utility that DP noise costs, as Feddle's ability to cancel corrupted directions suggests it might.

The main analysis concerns the mechanism, not leaderboard numbers. For a fixed atlas, we sample many coefficient vectors and compute the rank correlation between each guidance set's loss and the true word error rate on held-out client singing. The guidance sets are raw speech, unmatched songified speech, histogram-matched songified speech, and a small public real-singing set. If the hypothesis holds, raw speech will show weak or negative correlation and matched songification will show strong correlation. That would explain any end-to-end gains rather than merely report them.

Baselines

The naive reference is the public starting point with no federated training: PDAugment-style centralized pretraining, evaluated directly on client test data. PDAugment reports 9.8% WER on DSing30 and 54.0% WER on Dali (PDAugment). We rerun it in our own setup so the starting point is identical across all arms. Federated training that does not beat this model is not worth the system cost.

The strongest existing federated competitor is FedProx, which adds a pull toward the global model on each client to limit drift. It is the regularized averaging that FL-VetTrans needed for non-identical audio clients. We run it alongside plain FedAvg from the same initialization, with the same client data, round count and local epochs. Its proximal strength gets the same validation budget that our method gets for $\lambda$. No reported result on federated lyrics transcription was found on Lacuna, so the comparison numbers will be our own. The only reported number for the guided-versus-averaging gap is Feddle's 70.5% versus 52.6% on CIFAR-100. We treat that as evidence that merging can matter, not as an audio target. Our method counts as beating FedProx if it lowers WER on both DSing-derived and Dali-derived clients by a margin larger than the spread across three seeds, at equal communication.

The key ablation holds the merging procedure fixed and swaps only the guidance set. The options are raw LibriSpeech, songified speech with PDAugment's default score statistics, songified speech matched to client histograms, and a small public real-singing set. A second ablation removes the merge entirely and uses the matched songified set only as extra centralized pretraining. This separates the value of the guidance set from the value of the merge.

Why not just…?

Why not just keep training centrally on songified public speech and skip federation? This is the cheapest option, and it already reaches 9.8% WER on clean karaoke. The 54.0% Dali result in PDAugment shows that synthetic singing does not reach polyphonic, expressive real singing. GTSinger documents phonation techniques that pitch and duration edits cannot produce. Our second ablation, histogram-matched songification used as pretraining only, tests this option directly. If it matches the federated arms, we will report that client statistics alone are the useful signal and federation is unnecessary.

Why not just run FedProx and let the server average? This is the most serious threat. FedProx is simple, and if singing clients drift less than we expect, it may be enough. The evidence against it is indirect: Feddle beats FedAvg under drift and noisy clients, and FL-VetTrans needed regularization even on easy classification. Neither result is on sequence transcription. The head-to-head comparison at matched communication decides the question, and we report it however it comes out.

Why not just use a small public real-singing set as guidance? For English monophonic karaoke this may work as well as our method. If so, the songification step adds nothing in that regime. Our argument is about regimes where public singing does not match the client population, such as polyphonic, non-English, or technique-heavy singing. GTSinger's language and singer partitions let us hold out regimes with no matching public singing. The rank-correlation analysis shows where each guidance set stops tracking client error.

Why not just anonymize the singing on-device and upload it? SingFake reports speech-trained detectors at about 50% EER on singing. Bridging Speech and Singing finds that speech speaker embeddings transfer poorly to singing, with baseline speaker similarity of 0.448. Together these suggest that speech-built anonymizers would neither reliably remove a singer's identity nor preserve the lyric content. We do not evaluate this route, and we flag it as untested here rather than ruled out.

Proposed Experiments

The primary experiment compares WER after a fixed number of communication rounds across five arms on DSing30-derived and Dali-derived clients: the PDAugment starting point, FedAvg, FedProx, guided merging with raw-speech guidance, and guided merging with histogram-matched songified guidance. All arms use three seeds and identical client partitions. A GTSinger singer-partitioned run, scored with character or word error rate per language, tests expressive and multilingual clients, where public real-singing guidance is weakest.

The decisive mechanism check is the rank correlation between guidance loss and true held-out client WER over sampled merges. The proposal stands if matched songification gives clearly higher correlation than raw speech and if that ordering predicts the end-to-end gains. It fails if raw speech guidance works as well, since that would mean Feddle's out-of-domain robustness already crosses the gap. It also fails if FedProx matches guided merging, since that would mean the server-side merge is unnecessary.

Privacy experiments sweep the histogram budget $\epsilon$ and add DP-SGD on client updates at several noise levels, reporting WER against privacy budget for FedProx and for guided merging. We also run a membership-inference attack on the released models, measured as attack advantage over chance, to check that the histogram channel adds no measurable leakage beyond what the weight updates already carry.