In a tone language such as Mandarin, the pitch pattern of a syllable changes the word: mā and mà differ only in pitch. Singing takes over pitch to carry the melody, so it is unclear how much of that word-distinguishing pitch survives in a song. We propose to measure this with paired recordings. The same singer speaks and sings the same lyrics. We align the two recordings frame by frame with an alignment step that is not allowed to look at pitch, then ask where inside each sung syllable the spoken tone contour reappears. The working hypothesis comes from a field study of Kammu singing (Preservation of lexical tones in singing in a tone language), which found High/Low tone contrasts of 3–4 semitones at the vowel onset and melody taking over the rest of the syllable. Our hypothesis is that Mandarin singers also leave a brief tone "signature" near the start of each syllable, visible as a deviation from the note they are about to hold.
If that signature exists, it points to a cheap, specific input feature for recognizing lyrics in sung Mandarin: pitch measured relative to the syllable's own sustained note, taken only in an onset window. Current systems cannot see this signal cleanly. Absolute pitch is dominated by melody, and models built on self-supervised speech representations are not guaranteed to keep fine pitch detail after fine-tuning. The work has two parts. The first is an analysis of Mandarin tone survival in song, which extends the Kammu finding from a two-level register system to contour tones. The second is a feature that follows from that analysis, tested on the standard Mandarin singing benchmarks.
Core Claim
In Mandarin singing, the information that separates lexical tones sits in the gap between sung pitch and the held note, and it is concentrated near syllable onset. Giving a singing transcription model this onset-relative pitch reduces errors where the syllable is right but the tone-determined character is wrong. It should do so more than giving the model absolute pitch, and more than relying on the pitch the model encodes implicitly.
The dynamic time warping (DTW) component serves as the measuring instrument. DTW is an algorithm that finds the best monotonic frame-to-frame matching between two sequences played at different speeds. It is not the deployed system. At test time the transcription model sees only singing and needs neither a score nor paired speech.
What Already Exists and How This Differs
Three lines of work sit close to this proposal.
Phonetic segmentation of singing voice using MIDI and parallel speech is the closest in method. It aligns singing to the same singer's spoken lyrics with two DTW passes over spectral features (MFCCs) and reaches a 45.5 ms mean phonetic boundary error on four English songs. It motivates our alignment design but also constrains it. The data are English, so tone never arises, and the spectral features let pitch and vocal mode leak into the matching cost.
WhisperMSS is the strongest system on our target task. It adapts Whisper, a large pretrained speech recognizer, to Mandarin singing. It runs DTW over the decoder's attention matrices to obtain syllable timings and uses them to supervise a CTC branch, a frame-level output head trained to emit the correct label sequence without needing exact timings. It uses no explicit pitch.
Improving Lyrics Alignment through Joint Pitch Detection shows that predicting musical pitch helps locate syllable onsets. It treats pitch purely as note class and ignores lexical tone.
Our proposal differs from all three. It separates lexical tone from melody inside a sung syllable and uses that separation, not raw pitch, as the added signal.
On the tone-feature side, the file supplies the methods we reuse, along with evidence that implicit encoding is fragile. Pitch-Aware RNN-T fuses HuBERT features with a stream of pitch values sorted into discrete bins, reports 26.69% phone error rate on LATIC, and finds that binned pitch beats raw continuous pitch by 16% relative. It works on speech only, where absolute pitch is the tone. Encoding of lexical tone in self-supervised models shows wav2vec 2.0 encodes tone in middle layers, though English fine-tuning discards it. Tone in Perspective shows that flattening pitch badly damages recognition in tone languages (Thai WER +0.232). We also borrow its flattening manipulation for a different purpose: to make our alignment blind to pitch.
Conversion and augmentation work treats target pitch as a uniform note contour across the whole syllable. This includes AlignSTS, SVPT and PDAugment. Our analysis tests whether that assumption discards tone in Mandarin. We do not build a converter.
Research Program
Measurement on paired data. We use the Mandarin portion of GTSinger, which provides singing, the same singer's spoken lyrics, realistic scores and phoneme annotations.
Before extracting features, we resynthesize both recordings with pitch flattened to the utterance mean, following the manipulation in Tone in Perspective. We then take middle-layer self-supervised features from the flattened audio. The flattening prevents a circular result. If pitch entered the matching cost, the alignment would be pulled toward matching sung and spoken pitch, which would inflate the very correspondence we want to measure. The DTW is constrained to within-syllable windows, using the singing annotations and forced alignment of the speech, and its cost is normalized by path length.
We first validate alignment accuracy against the annotated phoneme boundaries. Dong et al.'s 45.5 ms serves as a rough reference only, since it comes from different data.
For each sung syllable we then compute the residual
$$r(t) = f(t) - \operatorname{median}_{\tau \in S} f(\tau),$$
where $f(t)$ is sung pitch in semitones at frame $t$ and $S$ is the last half of the syllable's voiced frames, which stands in for the held note. We compare $r(t)$ with the time-warped, mean-normalized spoken contour. We also train a small tone classifier on $r(t)$ in sliding windows at different relative positions within the syllable.
The hypothesis predicts two things. Tone accuracy should peak in an early window and fall toward chance in the held portion. Contour tones (rising tone 2, falling tone 4) and the level tone 1 may differ in how much survives, which the Kammu study could not test. Mandarin tone sandhi (context-driven tone changes) and neutral-tone syllables will be labeled by their surface tone and reported separately.
The feature. The onset window found in the measurement step fixes a single input stream: $r(t)$ inside the window, masked elsewhere, plus a voicing probability. Following Pitch-Aware RNN-T, we sort it into bins and embed it. Estimating the held note from the syllable's own later frames means neither a score nor a melody model is needed. Syllable extents at inference come from the model's own CTC segmentation.
The model. We start from a WhisperMSS-style recipe on Opencpop and on M4Singer, which has 20 singers across the SATB range. The only change is fusing the pitch stream into the encoder output with a small attention-plus-convolution block like Pitch-Aware RNN-T's.
Baselines
Do-nothing reference. This is our reproduction of the WhisperMSS recipe: Chinese character tokenizer, CTC branch and attention-DTW pseudo-labels, trained on the same data splits with the same compute. WhisperMSS reports 4.6% CER, 1.1% SyER, 0.020 s syllable boundary error and 92.3% onset F1 on Opencpop. Our reproduction must land near these numbers before any comparison counts. On M4Singer, no reported transcription CER was found on Lacuna, so our reproduction defines the reference there.
Absolute pitch, concatenated. The pitch value, its first and second time derivatives, and voicing probability are appended to the encoder input, as in Large vocabulary Mandarin speech recognition with different approaches in modeling tones.
Absolute pitch, fused. Binned absolute pitch is embedded and passed through the same fusion block, as in Pitch-Aware RNN-T. This is the strongest existing way to add tone and the main competitor.
All three pitch variants receive the same pitch tracker, the same number of added parameters, and the same hyperparameter search budget.
Ablations isolating the key ingredient. The first applies the onset-relative residual over the whole syllable instead of the onset window. The second uses a window of the same length at a randomly placed position. The third keeps the onset window but uses absolute pitch.
What counts as beating the baselines. Onset-window $r(t)$ must reduce tone-confusion character errors relative to both absolute-pitch baselines and the whole-syllable variant, with a paired bootstrap interval over utterances that excludes zero. It must also leave syllable error rate and boundary error unchanged within noise.
Why not just…?
Why not trust Whisper's implicit pitch, since WhisperMSS already reaches 4.6% CER? This is a real threat. The headline number leaves little room. But the file reports CER at 4.6% against SyER at 1.1%. If SyER is scored on toneless syllables, most character errors have the right syllable and the wrong character, which is exactly where tone or context decides. Our first check splits that gap into tone-differing confusions and same-tone homophones. If tone-differing errors are a negligible share, we stop the modeling part and publish the measurement study alone. The evidence that fine-tuning can discard tone (Shen et al.) makes the implicit route plausible to fail but does not prove it. The experiment has to show it.
Why not just add absolute pitch the standard way? In speech, absolute pitch is the tone. In song, it is mostly melody. Pitch-Aware RNN-T's gains were measured only on speech. The absolute-fused baseline tests directly whether a strong model can recover the tone residual from absolute pitch by itself. If it matches our feature, the onset hypothesis adds nothing for recognition, and we will report that.
Why not fix homophones with a stronger lyrics language model? Context can resolve many tone confusions without any acoustics. We will run every system with and without the same language-model rescoring. The claim survives only if the pitch-stream gain on tone-differing errors persists after rescoring.
Why not skip DTW and map spoken to sung syllables linearly using the annotations? A sung syllable is a short consonant followed by a stretched vowel, so linear mapping places the spoken onset contour too late. We run the measurement with both mappings. If they yield the same onset profile, we will say DTW was unnecessary for this analysis.
Decisive Checks
We run the checks in order.
The first is the error decomposition of the reproduced WhisperMSS on Opencpop and M4Singer. It decides whether the modeling part is warranted.
The second is the GTSinger measurement. Against the pitch-blind alignment, the hypothesis predicts an early-window tone accuracy clearly above chance that decays across the held portion. A flat profile, meaning tone evenly present or absent throughout, falsifies the onset account for Mandarin. Either outcome is reportable.
The third is transcription on Opencpop and M4Singer. Metrics are CER, SyER, a tone-confusion error rate computed from the character-to-pinyin mapping, and syllable boundary error, compared across the baselines and ablations above. Evaluation on M4Singer is split by voice type, which tests whether the feature holds across registers where pitch tracking is harder.
No reported tone error rate on sung Mandarin was found on Lacuna, so this metric will be established by our reproduced baseline.
References
- Preservation of lexical tones in singing in a tone language
- Phonetic segmentation of singing voice using MIDI and parallel speech
- WhisperMSS: A Two-Stage Framework for Mandarin Singing Transcription and Segmentation Using Pretrained Models
- Improving Lyrics Alignment Through Joint Pitch Detection
- Pitch-Aware RNN-T for Mandarin Chinese Mispronunciation Detection and Diagnosis
- Encoding of lexical tone in self-supervised models of spoken language
- Tone in Perspective: A Computational Typological Analysis of Tone Function in ASR
- AlignSTS: Speech-to-Singing Conversion via Cross-Modal Alignment
- Self-Supervised Singing Voice Pre-Training towards Speech-to-Singing Conversion
- PDAugment: Data Augmentation by Pitch and Duration Adjustments for Automatic Lyrics Transcription
- GTSinger: A Global Multi-Technique Singing Corpus with Realistic Music Scores for All Singing Tasks
- Large vocabulary Mandarin speech recognition with different approaches in modeling tones