Surprise That Ignores Tempo and Register: Phrase Boundaries in Sung Pitch Contours

A melody sounds like it has "phrases": units that start, develop and end. One way to find them without labels is to train a model to predict what comes next and to mark a boundary wherever its predictions suddenly fail. This works on written music, where notes are clean, evenly spaced tokens. It breaks on real singing. There the input is a continuous pitch curve, and singers stretch held notes, slide between pitches and move a phrase up an octave or a fifth. A next-frame predictor is "surprised" by every one of these changes, so its error spikes at changes of pace and register rather than at changes of musical content.

We propose to measure prediction error in a way that cannot see those nuisances. The model forecasts a short window of the pitch curve. Its error is the distance between forecast and reality after the best small time-warp and the best octave, fourth or fifth shift. Only errors in shape survive this comparison, and those spikes mark the boundaries. The same boundaries then replace brute-force sliding windows in large-scale motif mining, which has needed trillions of alignment computations.

Core Claim

Three pieces of earlier work each hold part of this idea. Unsupervised Symbolic Music Segmentation using Ensemble Temporal Prediction Errors shows that next-note surprisal (negative log-likelihood) from an ensemble of recurrent networks marks phrase boundaries in monophonic folk songs, reaching an unsupervised F-score of 77.03 on the Essen collection. It works on quantized scores with absolute pitch, and it needs pause and bar-line heuristics plus ensemble voting to suppress false spikes.

Two studies, An evaluation of methodologies for melodic similarity in audio recordings of Indian art music and Improving Melodic Similarity in Indian Art Music Using Culture-Specific Melodic Characteristics, establish what is needed to compare sung contours. Dynamic time warping (DTW), which aligns two sequences by stretching time locally, clearly beats rigid Euclidean comparison. The right pitch normalization depends on the tradition: tonic-relative pitch works best for Hindustani music, while subtracting the mean works best for Carnatic music, because Carnatic motifs recur in different registers. Normalizing for shifts of a fourth, fifth or octave, and truncating long sustained notes before alignment, both help further. Both studies assume the phrase boundaries are already given.

Our claim is that these comparison tools belong inside the error signal of a self-supervised predictor, not in a separate stage after segmentation. Concretely, let $x_{t:t+H}$ be the observed tonic-relative pitch curve (in cents) over a horizon of $H$ frames after time $t$, and let $\hat{x}_{t:t+H}$ be the model's forecast from the preceding context. We define the surprise at time $t$ as

$$s_t = \min_{k \in K} D^{(b)}_\gamma\big(\hat{x}_{t:t+H} + k,\; x_{t:t+H}\big),$$

where $D^{(b)}_\gamma$ is soft-DTW restricted to a warping band of width $b$ with smoothing $\gamma$, and $K = \{0, \pm 500, \pm 700, \pm 1200\}$ cents is the set of allowed fourth, fifth and octave shifts. Soft-DTW is the differentiable "soft-min" relaxation of DTW used in D3TW. Boundaries are peaks of $s_t$ above a locally adaptive threshold.

The prediction is specific. A held note or a moved register should produce almost no surprise. A new melodic gesture should produce a spike.

Research Program

The predictor is a small causal sequence model over pitch-tracked audio sampled at about 50 Hz, the middle of the sampling-rate range tested in the Indian art music evaluation. It is trained to forecast $H$ frames, roughly 0.5 to 1 second, using the band-limited soft-DTW loss. The transposition minimum is applied only when computing surprise at scoring time. We will also test whether including it during training helps.

The band constraint matters. D3TW documents a failure it calls the "degenerated sequence problem": alignment costs that are minimized freely learn to collapse or skip segments. In our setting the equivalent failure is a forecaster that predicts a flat line and relies on the warp to absorb any motion. Bounding the warp to a fraction of $H$, and adding an ordinary squared-error term with a small weight, are the two guards we will test against this.

We will analyse two things beyond boundary accuracy. First, we decompose surprise spikes by cause: tempo change, register shift, sustained note, or annotated boundary. This checks whether each part of the invariance removes the false spikes it was designed to remove. Second, we use the detected boundaries to generate candidate segments for the DTW motif-mining pipeline of Mining Melodic Patterns in Large Audio Collections of Indian Art Music, replacing its sliding-window seeds. We measure retrieval quality and the number of DTW computations.

Baselines

The naive reference places boundaries at random or at evenly spaced positions, at the same rate per minute as each method's output. It is scored with the same tolerance windows. Any learned method must clear it by a wide margin, since loose tolerances flatter all detectors.

The rule-based competitor marks boundaries at silences and breath gaps and at the ends of steady notes longer than the truncation threshold of 0.5 s for Hindustani and 1.0 s for Carnatic music, taken from Improving Melodic Similarity in Indian Art Music Using Culture-Specific Melodic Characteristics. It encodes the musicological principle that phrases often end on sustained resting notes (nyas svar). It is therefore the most likely cheap method to rival ours on Indian art music. Its thresholds get the same tuning budget on the same development split as our method's threshold, band width and horizon.

The strongest existing method is the ensemble surprisal approach of Unsupervised Symbolic Music Segmentation using Ensemble Temporal Prediction Errors, which reports F 77.03 and R-value 81.53 on Essen. On audio we run it on contours quantized into note sequences, giving it both absolute and tonic-relative pitch so that tonal normalization is not our private advantage.

The strongest simple continuous competitor is a predictor of the same size, trained with squared error, run on contours that already have sustained notes truncated and the mean subtracted. This baseline captures most of what our invariances aim at, but does so as a fixed preprocessing step. It is the key ablation.

Further ablations remove the transposition minimum, remove the warp (setting $b=0$), and swap tonic-relative for mean-subtracted input. Together these show which ingredient carries any gain.

We count it as beating the baselines if our method exceeds the truncated, mean-subtracted squared-error predictor and the quantized ensemble on boundary F-score, by a margin outside the confidence intervals across held-out recordings, on both Indian art music datasets. On clean symbolic Essen, it should stay within a few points of 77.03.

Why not just…?

Why not just preprocess the contour (truncate held notes, subtract the mean) and use an ordinary predictor? This is the real threat. If it matches us, the contribution reduces to a preprocessing recipe, and we will report that outcome. We expect it to fall short for three reasons. Truncation uses a fixed threshold, while singers vary their pace continuously. Mean subtraction over a window cannot handle a register shift that happens partway through that window. Neither step deals with glides. The ablation and the perturbation study described below decide the question directly.

Why not quantize the contour into notes and reuse the symbolic surprisal method? Quantization throws away the ornaments, the gamakas, which carry phrase identity in these traditions. It also introduces transcription errors exactly at slides, which is where boundaries tend to lie. The quantized ensemble baseline measures this cost. If it wins, the right conclusion is that better transcription, not warp-invariant surprise, is the bottleneck.

Why not rely on silences and long notes? Melodic Phrase Segmentation By Deep Neural Networks argues that long-note and rest heuristics miss subtler boundaries. On Essen, the rule-based ΔIOI compound baseline reached 75 against 77.03 for surprisal, a modest gap. The rule baseline is therefore run in full. We specifically report boundaries that do not coincide with a pause or a sustained note, since this is where any advantage of our method must come from.

Why not train a supervised detector? Supervised detectors do score higher where dense labels exist: about 84 F on Essen, and 85.83% F1 for a Bi-LSTM-CRF on Chinese pop songs. The Indian art music datasets, however, have only 157 and 340 annotated fragments. We include a supervised detector trained on those fragments as an upper reference, not as the target.

Proposed Experiments

The first experiment is a controlled perturbation study on Essen. Melodies are rendered as continuous contours. Each is then subjected to random monotone time warps, stretched held notes, synthetic glides, and within-piece shifts of an octave or a fifth, at graded strengths. The decisive check is whether our F-score stays flat as perturbation grows while the squared-error predictor and the symbolic ensemble degrade. On the unperturbed melodies we compare against the reported 77.03.

The second experiment uses real audio. On the Hindustani and Carnatic datasets, the annotated fragment endpoints serve as partial ground truth, scored with tolerance windows of 100 to 300 ms. Precision is computed only inside annotated regions. No reported boundary-detection result was found on Lacuna for either dataset, so all baselines are run by us. We also use GTSinger, which pairs singing with realistic scores. From it we derive phrase boundaries using rests and lyric lines in the aligned score, which provides a test across nine languages beyond Indian traditions.

The third experiment measures downstream retrieval. Our boundaries seed the mining pipeline, and we report retrieval MAP against the published reference points: 0.552 on the Hindustani dataset with tonic normalization and DTW, 0.413 on the Carnatic dataset with mean subtraction and DTW (An evaluation of methodologies for melodic similarity in audio recordings of Indian art music), and 0.92 on the S1 seed category of the full corpus (Mining Melodic Patterns in Large Audio Collections of Indian Art Music). We also report DTW computation counts. Boundary seeds that match sliding-window MAP while using far fewer computations would be a practical result even if boundary F-scores are only on par with the baselines.