Aligning Anyone's Speech to Anyone's Singing

When a person sings a lyric, the words are the same ones they would speak, but the sound is stretched and bent. Vowels last several times longer, pitch sweeps across an octave, and vibrato wobbles the spectrum. If we could reliably map each moment of a spoken reading onto the matching moment of a sung performance, we would get three things. We could transfer phoneme boundaries from easy-to-label speech to hard-to-label singing. We would have a direct measurement of how speaking rhythm turns into musical rhythm, which speech-to-singing converters currently fake. And we would have a diagnostic for exactly where speech models lose track of singing. The classic tool for this kind of mapping is dynamic time warping (DTW): an algorithm that finds the cheapest monotonic path pairing frames of one sequence with frames of another, given a frame-to-frame distance. The catch is that the only published speech-to-singing DTW system worked only when the speaker and the singer were the same person.

This note proposes to make speech-to-singing warping work across speakers. The plan is to learn a small projection on top of a frozen self-supervised speech model, trained with a differentiable DTW loss so that sung and spoken versions of the same words land close together even when they come from different people. The warping path is then scored as an object in its own right, by how accurately it transfers word and phoneme boundaries onto held-out singers. It is compared head-on against the simpler route of aligning speech and singing to text separately and composing the two maps.

Core Claim

Frame representations taken from a middle layer of a self-supervised speech model and lightly adapted with a contrastive soft-DTW objective on paired speech and singing will support cross-speaker speech-to-singing alignment. The expected boundary error should be close to what same-speaker alignment achieves. The adapted features should also beat MFCC warping, coarse phonetic-class warping, frozen self-supervised features, and text-mediated forced alignment. The gains should concentrate where singing departs most from speech: long sustained vowels, high registers, and marked techniques such as falsetto and breathy voice.

What Prior Work Establishes

The motivating result is Phonetic segmentation of singing voice using MIDI and parallel speech. It transferred phoneme boundaries from a singer's spoken reading of the lyrics to their sung rendition using DTW over MFCCs, reaching a 45.5 ms mean boundary error. The authors report that the method degrades badly when speaker and singer differ, because timbre differences dominate the MFCC distance. It also needed a MIDI score for coarse word anchoring. Our proposal keeps the core idea of "speech as a Rosetta Stone" and removes both the same-speaker requirement and the score.

The representation choice comes from speech-variation work. Neural representations for modeling variation in speech ran DTW over hidden layers of wav2vec 2.0. That DTW distance correlated with human accent ratings at r = -0.87, against r = -0.71 for MFCC-based DTW, which suggests middle layers encode phonetic content while discarding speaker and channel. SpeechBERTScore similarly found that middle-to-late layers of HuBERT and WavLM carry phonetic identity robustly. Both studies stayed inside speech, however. Nothing in the file tests whether these layers survive the jump to singing. The VoiceMOS Challenge 2024 is a warning here: speech-trained quality models failed on singing.

A second constraint comes from A Real-Time Lyrics Alignment System Using Chroma And Phonetic Features For Classical Vocal Performance. Its phonetic posteriors trained on TIMIT worked better on operatic singing only after being collapsed to five broad classes. That is direct evidence that fine-grained speech phonetic features become unreliable under singing. It is also why we do not expect frozen features to suffice. That system aligns singing to a singing reference, not to speech.

The training loss comes from Soft-DTW: a Differentiable Loss Function for Time-Series. It replaces the hard minimum in the DTW recursion with a smoothed log-sum-exp, which lets gradients flow through the alignment. Two existing speech-to-singing lines handle timing without any learned alignment. Self-Supervised Singing Voice Pre-Training towards Speech-to-Singing Conversion simulates speech rhythm through random resampling of singing. PDAugment stretches vowels of forced-aligned speech to fit MIDI note durations. Bridging Speech and Singing aligns only global speaker embeddings across the two domains, not frames. None of these produces a measured frame-level map between real speech and real singing, which is what this work builds.

Research Program

The method has three parts.

Representation and loss. We extract frames from a chosen layer $\ell$ of a frozen model, starting with WavLM and wav2vec 2.0 XLSR-53 and sweeping $\ell$. We add a two-layer projection $f_\theta$. The training pairs are a spoken line $x$ and a sung rendition $y$ of the same lyric line, deliberately drawn from different people. The loss is

$$\mathcal{L}(\theta) = \mathrm{sDTW}_\gamma(f_\theta(x), f_\theta(y)) - \log \sum_{y'} \exp\!\big(-\mathrm{sDTW}_\gamma(f_\theta(x), f_\theta(y'))/\tau\big),$$

where $\mathrm{sDTW}_\gamma$ is the soft-DTW cost with cosine frame distance and smoothing $\gamma$, $y'$ ranges over the true sung line and sung lines with different lyrics in the batch, and $\tau$ is a temperature. The contrastive term is needed because minimizing soft-DTW alone can be satisfied trivially by collapsing all frames to one point.

Warping constraints. At inference we use hard DTW with a step pattern that allows long, cheap runs on the singing side. This lets one spoken vowel frame absorb many sung frames. Consonant regions keep near-diagonal slopes, using a cost term that depends on the local sung energy and voicing. The intent is to handle the degenerate many-to-one paths that plain DTW produces under large tempo differences, one of the open problems in the DTW literature, without making them free everywhere.

Evaluation of the path. Word and phoneme boundaries annotated on the spoken side are projected through the path onto the singing, and the error is measured against gold singing boundaries. Training uses NHSS (10 professional singers who each read and sing 100 pop songs) and GTSinger (paired speech, phoneme-level annotations, and technique labels across 9 languages). Cross-speaker pairs are built by matching one person's reading with another singer's rendition of the same song where overlaps exist, or by pairing a GTSinger speaker with a different singer of the same song. Singers in the test split never appear in training on either side.

Baselines

The naive reference is a line-level linear stretch. Each spoken line is uniformly time-scaled to the sung line's length, given line boundaries, and boundaries are mapped proportionally. Every method gets the same line boundaries, so any method that fails to beat this one has learned nothing about within-line rhythm.

The classical baseline reproduces the second DTW stage of Dong et al.: MFCC plus energy features and standard DTW. It is run once same-speaker, where NHSS allows it, and once cross-speaker. Dong et al. report a 45.5 ms mean phonetic boundary error in the same-speaker setting on their own 4-singer corpus. That number anchors expectations but is not directly comparable to our test sets. The coarse-class baseline replaces features with five-class phonetic posteriors, following Park et al., and adds their chroma features on the singing side where applicable.

The frozen-feature baseline runs DTW on the best raw self-supervised layer with no projection, in the manner of Bartelds et al.. The layer is selected on the validation split, so it gets the same selection budget as our model.

The strongest competitor is text-mediated composition. A speech forced aligner, the Montreal Forced Aligner as used in PDAugment, timestamps the speech. A speech aligner fine-tuned on NHSS and GTSinger singing training data timestamps the singing. The two text-indexed maps are then composed. The singing aligner gets the same training data and a comparable tuning budget to our projection. The reference number for this family is the NHSS paper's model-based alignment of singing, with a median word boundary deviation of 0.072 s (NHSS). For polyphonic lyrics alignment, Improving Lyrics Alignment through Joint Pitch Detection reports an average absolute error of 0.23 s on Jamendo, but that is a different task. Beating composition means a lower median word boundary deviation on held-out NHSS singers than both the composed aligner and the 0.072 s figure. It also means lower phoneme-boundary error on GTSinger, with a paired bootstrap over lines showing significance.

Two ablations isolate the key ingredients. The first replaces the soft-DTW term with a frame-wise contrastive loss on frames paired by the text-mediated alignment. This tests whether learning through the warp matters or whether any cross-domain supervision works. The second trains on same-speaker pairs only, which tests whether cross-speaker pairing is what buys speaker invariance. Varying $\gamma$ over a small grid is reported rather than tuned per test set.

Why not just…?

Why not just align both signals to the lyrics text and compose? This is the most serious threat, and it is the strongest baseline above. It needs no parallel speech at all. Its weakness is that the singing side relies on an aligner whose speech-derived duration priors are exactly what extreme vowel elongation breaks, as Dong et al. discuss. Composition also inherits the errors of two aligners. If composition matches our method on NHSS and GTSinger, we will report that the text route suffices for boundary transfer. The remaining contribution would then be limited to the no-transcript setting: speech-to-singing matching for unwritten or low-resource lyrics, tested on the GTSinger languages where a forced aligner is weakest.

Why not just use frozen self-supervised features with plain DTW? This option is attractive because Bartelds et al. showed such features are already speaker-robust within speech. Park et al. found fine phonetic features unreliable on singing, however, so we expect frozen layers to fail specifically on sustained vowels and high registers. The frozen baseline and the per-technique breakdown settle this directly. If the projection adds nothing, the paper becomes a layer-wise map of where speech features break under singing, which is still useful.

Why not just songify speech with a vocoder and align synthetic to real? PDAugment shows vocoder songification helps transcription. It requires a MIDI score, though, and it imposes durations rather than measuring them, so it cannot tell us how a real singer stretched real speech. We include it only as a training-data source in a side experiment: songified speech replaces real sung partners in $\mathcal{L}$. If real pairs are no better than songified ones, parallel data collection is unnecessary.

Decisive Checks

The central test compares cross-speaker boundary error against same-speaker boundary error for each method on held-out NHSS singers. The claim holds if our method closes most of the cross-speaker gap that breaks MFCC warping and also beats text-mediated composition on median word boundary deviation. The second test breaks GTSinger phoneme-boundary error down by technique label (falsetto, breathy, vibrato-heavy) and by vowel duration. We expect the advantage over frozen features to grow with sung-to-spoken duration ratio; if it is flat, elongation is not the mechanism we are fixing. The third test is out-of-language transfer: training on English and Chinese, testing on other GTSinger languages. This checks whether the projection learned phonetic invariance or memorized a phoneme inventory. As a downstream sanity check, warping paths from test singers are used as duration targets when resynthesizing with the NHSS speaker-independent conversion setup. The resulting mel-cepstral distortion is compared to the reported 6.85 dB, which shows whether better alignment translates into better converted singing.