Corrections That Travel: Interactive Multi-Instrument Transcription with an Editable Recurrent State

When several instruments play at once, their sound energy lands in the same frequency bands. Automatic music transcription, the task of turning audio into a list of notes with an instrument label on each note, breaks down under this overlap. In the 2025 AMT Challenge, the best system scored an F-measure of 0.7193 on solo instruments and 0.4585 on mixtures of up to three. The organizers attribute the drop to "instrument leakage" (assigning notes to instruments that are not playing) and to pitch confusion. A musician who hears the result can usually tell at once what went wrong, for example "that low line is the cello, not the bassoon." No current system can take that sentence plus one click and use it to repair everything the mistake caused downstream.

This note proposes a transcriber built around a recurrent network, that is, a network that reads the audio frame by frame and carries a compact memory (its hidden state) forward in time. A user correction, given as a click on a note and optionally a short spoken or typed phrase, is written into that memory at the frame where it occurs. The network then re-decodes forward from that point. The core claim is that one correction should fix many errors, not one. Telling the model that a note at 0:14 is cello should also fix the following notes in that phrase that the model had given to bassoon, because the recurrent state carries "which instrument is sustaining which line" through time. The corrected note list is then passed through a fixed, swappable interface to a separation module, so better transcription directly yields cleaner isolated stems. Success is measured the way interactive image segmentation measures it: how many interactions it takes to reach a target accuracy.

Core Claim

A transcriber trained with simulated corrections as extra inputs, whose recurrent state tracks per-instrument note state (onset, sustain, decay), will need substantially fewer interactions to reach a target note-plus-instrument F1 on multi-instrument audio than either of two alternatives. The first is editing notes locally, where each click fixes only the note it touches. The second is re-running an automatic system with the corrections as hard constraints. Because only the frames after the correction are re-decoded, the recurrent design should also update fast enough to stay interactive. Whether recurrence itself is necessary, as opposed to simply training with simulated corrections, is a hypothesis the experiments will test directly. It is not assumed.

What Exists and What Is Missing

Three systems come closest, and each lacks one piece. Jointist supplies the method skeleton. It identifies which instruments are present, feeds that into a recurrent Onsets-and-Frames-style transcriber, and passes the resulting per-instrument piano rolls (time-by-pitch grids of active notes) to a U-Net separator, gaining 5 dB SDR on Slakh2100. It is fully automatic, though. It also reports a "cold start" problem in which noisy early transcriptions damage the separator during joint training. We reuse its three-module structure and add a correction pathway. That pathway also gives the separator cleaner rolls during training, which may ease the cold-start problem.

An Efficient Posterior Regularized Latent Variable Model for Interactive Sound Source Separation motivates the interactive premise. Users paint spectrogram regions belonging to each source, and a matrix-factorization model is re-fit within seconds as the user works. The painting is pixel-level and carries no meaning beyond "this region belongs to source A," and the model has no learned sense of time. We replace painting with clicks on notes and phrases about instruments, and we replace matrix factorization with a learned recurrent state.

DGMO separates sources from a text query using a diffusion model without retraining, reaching 1.89 SI-SDR on AudioCaps. It runs open-loop: once the user has given the query, there is no way to add a correction afterward.

Two further works shape the interaction and temporal design. PhraseClick supplies the interaction recipe. In image segmentation, a phrase says what and a click says where. The combination cut interactions to 85% IoU on PASCAL VOC from 6.88 to 3.12, and raised IoU on RefCOCO from 50.42% (phrase only) to 80.02% (phrase plus one click). We carry this pairing over to time and pitch. The phrase tells the model which instrument, and the click tells it when and at what pitch. Benetos & Weyde supply the temporal structure. Modeling notes as attack, sustain and decay states with enforced transitions helps separate instruments whose steady-state harmonics coincide but whose onsets differ, such as a hammered piano and a bowed violin. They used fixed hidden-Markov transition tables. We use learned gated recurrence, and we make those states the place where corrections are written.

Two papers constrain the design. LadderSym shows that a large audio-score transformer still drops notes masked by loud chords, at about 97 ms latency. So the interactive alternative to recurrence must be measured for latency, not assumed to be slow. MT3, a 60M-parameter sequence-to-sequence transcriber trained on six datasets, scores 0.3932 overall F1 on the challenge test set. This says scaling data alone has not solved overlap so far, although it does not prove that it never will.

Research Program

The system has three modules joined by explicit, versioned interfaces, in the style of the factorized design of Wave2Midi2Wave and the abstract-interface discipline of DeepDIVA and UCEE. The first module is the transcriber, which outputs per-instrument note-state rolls. The second is the correction encoder. The third is a separator conditioned on rolls. The interface between the transcriber and the separator is the piano roll itself. This lets us swap the recurrent transcriber for a transformer, or the learned separator for the matrix-factorization method, without touching anything else. That swap is what makes the ablations clean. Modularity here is a tool for fair comparison, not a contribution claimed for its own sake.

A correction event is a small record: a time frame, a pitch, an instrument label, and an action (add a note, delete a note, relabel its instrument, or "instrument X is not playing here"). A click on the displayed roll or spectrogram supplies the time and pitch. An optional phrase supplies the instrument and a coarse register ("the low line," "the flute"). A small text encoder maps the phrase to an instrument embedding plus a pitch-range mask, so a phrase alone can retarget a whole register. This asymmetric split follows Reinders et al., who found that people use a spatial overview to get their bearings and speech to pin down specifics. The events are injected as additional inputs to the recurrent transcriber at their frames, and also as hard constraints on its outputs at those frames.

The transcriber is a convolutional front end followed by gated recurrent layers. A forward pass carries state, and a bounded backward look-back window lets a correction also revise the frames just before it. Training uses simulated users, as in the click-simulation protocols used for PhraseClick-style evaluation. After an automatic pass, a simulator picks one of the model's largest errors, either the longest misattributed note run or a hallucinated instrument. It issues the corresponding event, sometimes with a templated phrase, and the loss is computed on the re-decoded output. In this way the model learns to propagate corrections rather than merely obey them locally. Training is on Slakh2100 (Cutting Music Source Separation Some Slakh) together with the real-recording training sets in the MT3 mixture.

Baselines

The do-nothing reference is the best fully automatic transcriber run once, with no interaction. For the challenge test set, the reported reference numbers are MIROS at 0.5998 overall F1 and 0.4585 multi-instrument F-measure, and MT3 at 0.3932 overall, all from the 2025 AMT Challenge. Our own model at zero interactions will be reported on the same footing. That number is not the claim, but it shows whether we start from a competitive automatic system or a weak one that interaction merely patches.

The strongest competitor on the actual question, how much accuracy each correction buys, is a transformer transcriber given the same correction events. It will be an MT3-style encoder-decoder of matched parameter count, trained with the identical simulated-correction curriculum and the same tuning budget. The correction events go in as prefix tokens and as decoding constraints, and the transformer re-decodes the affected segments. It is the strongest because it isolates whether recurrence matters, or whether training with corrections is all that matters. Beating it means a lower interaction count to reach the target F1 at comparable or lower update latency. If it matches us on interactions but is slower, the claim narrows to latency. If it matches on both, recurrence is not the ingredient and we will report that.

The local-edit baseline uses our own automatic output. Each correction changes exactly the note it touches, and nothing else moves. It measures the value of propagation directly. A further ablation keeps our recurrent model but trains it without simulated corrections, so that events act only as output clamps at inference. This isolates whether the model must learn to propagate or whether clamping the state is enough on its own.

On the separation side, the comparisons are as follows. The first is the Jointist-style separator fed automatic rolls. The second is the same separator fed rolls after k corrections. The third is the posterior-regularized matrix-factorization method of Bryan & Mysore, given an equal number of user actions (one brush stroke counted as one action). No reported decibel values for it were found on Lacuna, so we will rerun it. The fourth is DGMO given only the text query, which has no click pathway.

Why not just…?

Why not just correct the piano roll by hand and feed it to the separator? This is the local-edit baseline, and it is a fair lower bound on effort-free engineering. Its weakness is structural. Instrument leakage is a run of errors along a musical line, not an isolated mistake, so per-note fixing costs one interaction per wrong note. The propagation-gain measurement (errors removed per interaction) will show directly whether our model removes several per click on leakage-heavy passages. If it removes about one per click, the idea fails.

Why not just give a transformer the corrections as a prompt? This is the real threat. Nothing in principle stops an attention model from learning the same propagation, and LadderSym's 97 ms latency is not prohibitive. We do not argue around it. The matched transformer with the identical training curriculum is the main comparison, and the outcome decides whether the contribution is recurrence or only correction-aware training.

Why not just ask for the instrument by text and let a generative separator do the rest? DGMO-style querying handles "extract the cello" but has no way to say when the answer is wrong. Text-only control in the image domain topped out at 50.42% IoU against 80.02% once a single click was added (PhraseClick). The phrase-only and click-only ablations of our system, alongside DGMO with the text query alone, will show whether the same gap appears for overlapping harmonics.

Why not just scale an automatic transcriber? Scaling may eventually close the gap. For now, the challenge organizers name data scarcity as the bottleneck, and the best entrant still loses about a third of its accuracy on mixtures. Interaction is complementary to scaling. A stronger automatic start lowers the number of corrections needed, and our protocol reports that start explicitly.

Proposed Experiments

The primary evaluation uses simulated users on real multi-instrument recordings. These are URMP and Bach10, plus the 2025 AMT Challenge test set if its labels are available to us, with Slakh2100 held-out songs as a synthetic check. The headline metric is the number of interactions needed to reach note-plus-instrument F1 of 0.8, capped at 20 per piece. This is the music analogue of PhraseClick's interactions-to-85%-IoU. No reported interactive-transcription result was found on Lacuna for any of these datasets, so all interactive baselines are rerun. We also report F1 after 1, 3 and 5 interactions, errors removed per interaction, and wall-clock update latency against a 100 ms interactive budget. Separation SDR and SI-SDR on Slakh and URMP stems after k corrections test whether the gains reach the audio. For reference on Slakh, Jointist's reported gain from symbolic guidance is 5 dB SDR, though the file gives no absolute value.

The decisive checks are three. The first is whether propagation gain exceeds one error per interaction on leakage-heavy passages, where the local-edit baseline is one by construction. The second is whether the recurrent model beats the matched transformer on interaction count or latency. The third is whether the no-correction-training ablation loses most of the benefit. A small study with musicians on 20 excerpts will check that simulated-user rankings hold for real users, measuring interactions and time to an acceptable transcription. It follows the user-study pattern of PhraseClick, where real users went from 4.49 interactions with clicks only to 3.09 with phrases added.