Melody extractors are the systems that read the sung or played tune out of a full music mix. They often name the right note but in the wrong octave: they say "C" correctly while confusing C3 with C4. The cause is physical. A note at 220 Hz has overtones at 440, 660, and 880 Hz, and every overtone of the note one octave up coincides with an even overtone of the lower note. The spectrum alone therefore cannot always tell the two apart. The best recent fix splits pitch into two predictions, a pitch-class ("tone") head and an octave ("register") head. These two heads want opposite things from the shared network underneath them. The tone head is rewarded for features that ignore which octave a note is in. The octave head needs exactly the information the tone head is taught to discard.
This proposal claims that this tension is a specific, measurable kind of multi-task interference, and that it is local in time. The two heads' training signals should pull against each other mostly on the minority of frames where the fundamental is weak relative to its octave neighbours, and those are the frames where octave errors happen. If that holds, the general-purpose multi-task optimizers average the conflict away across a batch, which is the wrong treatment. A cheap per-frame rule should work better: on ambiguous frames, the tone head's gradient yields to the octave head's. We will first test whether the conflict exists and where it lives, and build the method only on what that measurement shows.
Background and the gap
Pitch trackers have long suffered "pitch halving" and "pitch doubling." This was reported for unsupervised piano transcription in Unsupervised Transcription of Piano Music and for harmonic-model melody estimation in Melody pitch estimation based on range estimation and candidate extraction using harmonic structure model. Melody Extraction from Polyphonic Audio Based on Particle Filter documents the resulting evaluation split: Raw Chroma Accuracy (RCA, correct pitch class ignoring octave) stays high, while Raw Pitch Accuracy (RPA, correct exact pitch) and Overall Accuracy (OA, which also scores voicing) fall. These papers motivate the work, because they locate the error in the register decision rather than the pitch-class decision.
TONet supplies the architecture we build on. It rearranges spectral bins so that bins sharing a pitch class sit next to each other (Tone-CFP), splits prediction into tone and octave branches, and fuses them. It introduces Raw Octave Accuracy (ROA) and reports ROA/RCA gains of 5–16% over its host backbones. Its two heads are trained with a fixed loss combination, and the paper never asks whether they compete for the shared features. That is the gap we target.
MTANet attacks octave errors differently, with fixed frequency sub-bands, dilated convolutions, and an auxiliary voicing branch trained by plain loss summation. It is the strongest reported system in the file. JEPOO is the closest multi-task precedent in music. It balances pitch, onset, and offset with a Pareto-modulated loss, but it keeps pitch as one monolithic output and does not measure octave errors.
The multi-task optimization literature supplies candidate tools, and it also constrains the premise. CAGrad chooses an update close to the average gradient that maximises the worst-off task's improvement. RotoGrad learns per-task rotations of the shared feature space and rebalances gradient magnitudes. Both act on whole-batch, parameter-level gradients.
ForkMerge directly contradicts the gradient-conflict story. It shows that conflicting gradients can coexist with better generalisation, and it attributes negative transfer to auxiliary tasks pulling parameters toward a shifted distribution. It fixes this by forking training into target-only and joint branches and interpolating them using validation scores. We take this objection seriously. Our first experiment is designed so that a ForkMerge-consistent outcome, with no harmful localised conflict, would be visible and would redirect the work.
Core Claim
In tone–octave factorised melody extraction, the tone and octave losses send opposing gradients into the shared features, and this happens predominantly on octave-ambiguous frames. Resolving the conflict only on those frames, with octave priority, reduces octave errors more than three alternatives: tuning static loss weights, whole-batch gradient surgery, or parameter-space merging.
To make "ambiguous" concrete, let $S_t(p)$ be the input salience of the combined frequency-and-periodicity representation (CFP) at pitch $p$ in frame $t$, and let $p_t$ be the labelled pitch. The ambiguity score is
$$a_t = \frac{\max\{S_t(p_t - 12),\; S_t(p_t + 12)\}}{S_t(p_t)},$$
which measures how strongly an octave neighbour competes with the true pitch. Values near or above 1 mean the spectrum alone favours the wrong octave.
Let $z_t$ be the shared feature vector at frame $t$. Write $g^{T}_t = \partial \mathcal{L}_{\text{tone}}/\partial z_t$ for the tone loss's gradient with respect to $z_t$, and $g^{O}_t$ for the octave loss's gradient. The proposed rule applies only when two conditions hold: the frame is ambiguous ($a_t > \tau$, for a threshold $\tau$), and the two gradients disagree ($\langle g^{T}_t, g^{O}_t\rangle < 0$). In that case the tone gradient loses its component along the octave gradient before it reaches the backbone:
$$\tilde g^{T}_t = g^{T}_t - \frac{\langle g^{T}_t, g^{O}_t\rangle}{\lVert g^{O}_t\rVert^2}\, g^{O}_t .$$
The rule is asymmetric because the tone task is already the easy one, as the high RCA shows. Its cost is a few vector operations per frame on activations, with no extra backward passes.
The rule differs from CAGrad and RotoGrad in where it acts and when. It works on per-frame activation gradients rather than batch-averaged parameter gradients, and only on frames selected by acoustic ambiguity. It differs from ForkMerge in operating inside each step rather than across parameter trajectories.
Research Program
The work proceeds in three stages, and each one gates the next.
Stage 1: diagnose. We train TONet with the MSNet backbone (TO-MSNet) under its standard loss. Every few hundred steps we log three quantities, each split by ambiguity stratum: per-frame cosine similarity between $g^{T}_t$ and $g^{O}_t$, their magnitude ratio, and how well a linear probe on $z_t$ recovers the octave label. The "invariance leakage" hypothesis predicts that octave probe accuracy on high-$a_t$ frames peaks early and then falls as the tone head pushes the backbone toward octave-invariant features. It also predicts that negative cosines concentrate in the high-$a_t$ stratum. If the cosines are not reliably negative there, or the probe does not degrade, the gradient-conflict premise fails, and the project pivots to the ForkMerge distribution-shift account.
Stage 2: build the method. If the diagnosis holds, we implement the frame-local projection and tune $\tau$ on validation data only.
Stage 3: test transfer. We apply the same diagnostic to MTANet's voicing auxiliary branch, which the file's Gap 3 leaves untested. This checks whether the localised-conflict picture holds for auxiliary pitch tasks beyond tone and octave.
Baselines
The naive reference point is TO-MSNet trained exactly as published, with fixed loss weights. For its absolute numbers the file gives OA of 82.6% on ADC2004 and 85.3% on MIREX05 (TONet). We found no reported absolute RPA, RCA, or ROA values for TO-MSNet on Lacuna, only the 5–16% relative gains. We will therefore re-measure all four metrics ourselves.
The strongest existing system is MTANet, which reports OA of 86.9% on ADC2004, 89.2% on MIREX05, and 74.6% on the 12-clip MedleyDB test set (MTANet). It is the strongest competitor because it targets octave errors with a different mechanism and currently leads on every shared benchmark in the file. We will retrain it on the same data so we can report RPA, RCA, and the octave gap alongside its published OA. Beating it means exceeding its OA on at least two of the three test sets while also reducing the octave gap, defined as RCA minus RPA.
We do not expect to beat MTANet because the TONet backbone is stronger. We expect it because the projection should transfer to MTANet itself. For that reason we also report MTANet plus a tone/octave head pair trained with and without the projection.
The multi-task optimizer baselines are CAGrad, RotoGrad, and ForkMerge, each applied to the same TO-MSNet. For ForkMerge, octave is the target task and tone is the auxiliary. These are the competitors that would make our contribution redundant.
The key ablations isolate the frame-local gating. We apply the same projection to every frame regardless of $a_t$. We apply it to a random subset of frames matched in size to the high-ambiguity set. We apply it symmetrically, letting either gradient yield.
Fairness. All arms train on MIR-1K plus the 35 MedleyDB vocal tracks used by TONet and MTANet. All share an identical epoch budget and the same number of hyperparameter trials (twelve per method, covering $c$ for CAGrad, the merge interval for ForkMerge, the loss weights, and $\tau$). All use three seeds. Because ADC2004 and MIREX05 contain only 12 and 9 clips, we report clip-level bootstrap confidence intervals and claim a win only when the interval on the octave gap excludes zero.
Why not just…?
Why not just tune the loss weights between the tone and octave heads? This is the most serious threat. If a well-searched static weight on the octave loss matches the frame-local rule, then the conflict is a magnitude problem that a single scalar solves. The Stage 1 diagnosis already separates the two cases: a magnitude imbalance appears as a skewed gradient-norm ratio with non-negative cosines, while true directional conflict appears as negative cosines. The weight-grid arm gets the same trial budget as our method. If it ties, we will report that weight tuning suffices and present the diagnostic as the contribution.
Why not just stop the tone head's gradient from reaching the backbone, or give each head its own backbone? This removes the conflict trivially. It also discards whatever the tone task contributes to octave decisions, and TONet's fusion results suggest that contribution is real. We include both variants at matched parameter count. If a separate octave backbone equals our method, sharing is not helping and the paper's message changes to "do not share."
Why not just smooth the octave output over time, for example with HMM decoding? Raphael's HMM transcriber in Automatic Transcription of Piano Music shows that state priors help, yet it still reached a 39% note error rate. Smoothing also fixes only brief register jumps, not sustained wrong-octave passages. We will measure the length distribution of octave-error runs and apply Viterbi smoothing on top of every arm. Since smoothing works after the network, it should combine with our training-time change rather than replace it, and a large gain when stacked would show the two address different errors.
Why not just use large pretrained audio features, as Sheet Sage does? Melody transcription via generative pre-training reaches F1 0.744 on RWC-MDB using Jukebox embeddings. That comes at a very large compute cost, and it relies on fixed-meter and equal-temperament assumptions that do not hold for frame-level singing contours. This is a different operating regime, and we do not claim to compete in it. We do note that the frame-local rule could be applied to heads trained on such features.
Decisive Experiments
The first decisive check is the Stage 1 measurement. It must show significantly more negative tone–octave cosine on frames with $a_t > 1$ than on frames with $a_t < 0.5$, and a declining octave probe on ambiguous frames. Without both, the method is not pursued.
The second check compares octave gaps. The frame-local rule should cut the octave gap on ADC2004, MIREX05, and MedleyDB more than CAGrad, RotoGrad, ForkMerge, and the tuned-weight grid, all under matched budgets. The improvement should be concentrated in the high-ambiguity stratum, while accuracy on low-ambiguity frames stays unchanged.
The third check is the random-frame ablation. Projecting on random frames, matched in count, must recover substantially less of the gain. Otherwise the benefit comes from generic regularisation rather than from targeting ambiguity.
As a sanity check that the effect is tied to octave confusion rather than polyphony, we repeat the stratified analysis on monophonic MDB-stem-synth. On that set CREPE already reaches 96.7% RPA (CREPE), and we expect few high-ambiguity frames and correspondingly small gains.
References
- Unsupervised Transcription of Piano Music
- Melody pitch estimation based on range estimation and candidate extraction using harmonic structure model
- Melody Extraction from Polyphonic Audio Based on Particle Filter
- Tonet: Tone-Octave Network for Singing Melody Extraction from Polyphonic Music
- MTANet: Multi-band Time-frequency Attention Network for Singing Melody Extraction from Polyphonic Music
- JEPOO: Highly Accurate Joint Estimation of Pitch, Onset and Offset for Music Information Retrieval
- Conflict-Averse Gradient Descent for Multi-task learning
- RotoGrad: Gradient Homogenization in Multi-Task Learning
- ForkMerge: Mitigating Negative Transfer in Auxiliary-Task Learning
- Automatic Transcription of Piano Music
- Melody transcription via generative pre-training
- Crepe: A Convolutional Representation for Pitch Estimation