Learning Song Structure from Ordered Section Lists

Music structure analysis means splitting a song into its sections (intro, verse, chorus, bridge) and naming each one. Current systems learn this from dense annotations that give exact start and end times for every section, and such data is small: the Harmonix Set has 912 songs, mostly 4/4 pop (The Harmonix Set). Much weaker descriptions are far cheaper to collect. A listener, a catalogue, or a caption can say "acoustic intro, two verse–chorus cycles, a guitar solo, a quiet outro" without giving any timestamps. This note asks how much of the timing can be recovered from that ordered list alone. The tool for this comes from weakly supervised image segmentation, where single-stage models learn pixel masks from image-level tags by refining their own pseudo-labels during training (Single-Stage Semantic Segmentation From Image Labels, WeCLIP). Music adds two constraints that images lack. Sections occur in a known order, and the same section repeats.

The proposal trains a small decoder on top of a frozen music audio encoder. The decoder labels every bar of a song with a section name and a boundary probability. Its only supervision is the ordered list of section names, aligned to the bars by a differentiable monotonic alignment. A second loss requires bars that sound alike to receive the same label. Section names are scored against text embeddings, so audio and language are matched at the level of sections rather than whole tracks. Existing music–language models such as MusCALL, MuLaP and CLaMP all align at the track level.

Core Claim

The central claim is testable on existing benchmarks. With a frozen encoder held fixed, a single-stage decoder trained only on timestamp-free ordered section lists should close most of the gap between a linear probe and full supervision on Harmonix. It should also beat a two-stage pipeline that first segments the song without supervision and then aligns the list to those segments. The two-stage comparison is the real test, and the experiments are built around it.

The closest prior work falls into three groups, and none does this.

Our hierarchy differs in two ways. Its parts are bars rather than equal time slices, and its wholes are the segments the model currently predicts.

Research Program

Input representation. Each song passes through a frozen MusicFM encoder using the extended-context, threefold-downsampled setup of Temporal Adaptation. The frame features are pooled onto a downbeat grid, one token per bar. The Harmonix finding that 81.1% of segments start on a downbeat makes bars a natural unit. The 10% that start on a beat-4 pickup are handled by a beat-level offset head that adjusts each boundary's position within its bar. Downbeats come from Harmonix annotations in an oracle condition and from an off-the-shelf beat tracker in the realistic condition.

Decoder and labels. A four-layer transformer over the bar tokens produces two outputs per bar: a boundary probability and an embedding. The embedding is scored against text embeddings of the section names in the list. Because labels are scored as text, new label vocabularies (SALAMI's, RWC's) can be used without retraining a classifier head.

Ordered-list alignment loss. Let a song have $T$ bars and an ordered list $y = (y_1, \dots, y_K)$. Let $p_t(c)$ be the decoder's probability that bar $t$ carries label $c$. The loss is

$$\mathcal{L}_{\text{order}} = -\log \sum_{\pi \in \mathcal{A}_{d}(y, T)} \prod_{t=1}^{T} p_t(\pi_t).$$

Here $\mathcal{A}_{d}(y,T)$ is the set of monotone expansions of $y$ onto the $T$ bars in which every list item covers at least $d$ consecutive bars. The sum is computed with a forward recursion, as in connectionist temporal classification, so the loss is exact and differentiable. The minimum duration $d$ blocks the degenerate solution in which one item absorbs the whole song. It plays the role that tightness priors play in box-supervised segmentation (Bounding Box Tightness Prior).

Repetition consistency loss. This is the music counterpart of online affinity learning (Adaptive Affinity Loss). Pairs of bars that are confidently similar in the frozen-feature self-similarity matrix, or confidently dissimilar, are used as targets. The decoder's label posteriors for those pairs are pulled together or pushed apart. As training proceeds, the set of confident pairs is recomputed from the decoder's own sharpened posteriors. This lets two choruses far apart in time share a label even when the list is ambiguous. It also addresses the over-smoothing concern raised by ToCo: repetition, rather than local contrast, decides which bars should look alike.

Part–whole layer. Bar tokens attend to segment tokens, and segment tokens attend to bar tokens, in the bidirectional style of MART. Segment tokens are formed by pooling bars between the currently predicted boundaries. For SALAMI, the same layer is applied twice to produce an upper and a lower level of structure.

Supervision ladder. Weak labels are derived from the dense annotations of Harmonix and SALAMI, so every supervision level can be scored against ground truth on the same songs. The ladder runs from an unordered bag of section names, to an ordered list, to a list degraded in ways that mimic real captions (items dropped, adjacent items merged, labels swapped for synonyms), up to full timings. We do not have access to the private caption libraries behind MusCALL and MuLaP. Free-form captions are therefore out of scope for the core claim, and the degraded lists stand in for them.

Baselines

Naive reference. A linear probe on frozen MusicFM features. Do Foundational Audio Encoders Understand Music Structure? reports HR3F of 63.91% and function accuracy of 68.13% on Harmonix for this setup. Note that this probe is trained with dense labels, so it is a reference for backbone quality, not a lower bound on weak supervision. We also include a true do-nothing baseline that uses no training: each list item is spread uniformly across the song's bars in order.

Upper reference. The fully supervised temporal-adaptation model (Temporal Adaptation), which reports frame accuracy 0.787 and HR3F 0.828 on Harmonix. We rerun it with our exact data splits, a frozen backbone, and the same decoder size. The reported figures come from fine-tuning, so the frozen version gives the fair ceiling. Success means recovering at least two thirds of the gap between the linear probe and this ceiling, using ordered lists only.

Strongest competitor: two-stage alignment. Segments are first computed by spectral clustering on the hierarchical embeddings, using the same frozen MusicFM features as input. The ordered list is then aligned to those segments by dynamic programming over segment–text similarity. On SALAMI, the embeddings paper reports L-recall of about 0.68–0.69, against 0.664 for agreement between human annotators. For clustering-plus-alignment on Harmonix function labels, no reported result was found on Lacuna. The clustering hyperparameters receive the same validation-tuning budget as our model. Beating this baseline means a statistically significant gain in frame accuracy and HR.5F across five seeds.

Bag-of-tags baseline. A class-activation-map-style decoder trained on unordered tags with the same refinement loop. This isolates the contribution of order.

Ablations. Four ablations isolate the remaining ingredients:

  • removing the repetition loss;
  • replacing predicted segments with fixed halves and quarters in the part–whole layer;
  • replacing the bar grid with uniform frames;
  • swapping the text-embedding classifier for a fixed classifier head, which tests whether matching against language matters for transfer to new label sets.

Why not just…?

Why not just slide an audio–language model like CLAP over the song with prompts such as "chorus"? The linear-probing study Do Foundational Audio Encoders Understand Music Structure? found that CLAP lags well behind masked-modeling encoders on boundary detection and function labeling. MusCALL pools attention over the whole track. We still run sliding CLAP prompts as a zero-shot baseline, but we expect them to fall below even the linear probe.

Why not just segment without supervision and then attach the list? This is the real threat. If unsupervised boundaries are already good, aligning the ordered list afterward might capture almost all the value, and our single-stage design would add little. The two-stage alignment baseline tests this directly, with matched tuning budgets. If it ties our model, the conclusion becomes that the ordered list is the useful signal and joint training adds nothing. We would report that result as such.

Why not just fine-tune on a few densely labeled songs? That is a fair comparison only at equal annotation cost. We will time how long annotators take to produce an ordered list versus exact timings for a set of songs, then compare at equal annotator-hours. We train the supervised model on $k$ densely labeled songs and our model on the larger number of list-labeled songs that the same time buys.

Why not just use the classic class-activation pipeline with a separate refinement step? The image literature documents that errors in the first-stage seeds cannot be recovered later (ECS-Net). Moreover, tags alone carry no order information, so this pipeline cannot use the signal the proposal depends on. The bag-of-tags baseline measures both effects.

Proposed Experiments

Main comparison. We train on Harmonix with five-fold cross-validation and test on RWC-Pop without adaptation. Metrics are HR.5F, HR3F and frame-wise function accuracy.

Hierarchy. On SALAMI, we supervise the upper level with lists of the coarse sections and report the L-measure for both levels. This shows whether lower-level structure emerges from the part–whole layer without any lower-level labels.

Degraded lists. We measure how accuracy falls as list items are dropped, merged, or renamed with synonyms. A slow decline would suggest the method can survive real captions. A steep decline would suggest that real captions need cleaning before use.

Decisive checks. The outcome turns on two results. The first is whether the ordered-list model beats two-stage alignment on Harmonix frame accuracy. The second is whether the repetition loss recovers labels for sections that the list mentions only once but that occur several times in the song.