Harmonic Refractoriness: A Cross-Pitch Recovery Kernel for Polyphonic Transcription

Music transcription systems turn audio into notes. They fail most often when several instruments play at once. When one note sounds, its overtones land on the same frequencies as other possible notes, an octave or a twelfth above it. As that note decays, or as a neighbouring instrument starts, the fluctuating energy makes the detector fire a second, spurious note at a harmonically related pitch or on the wrong instrument track. Neurons face a structurally similar problem and handle it with refractoriness: after firing, a neuron's threshold rises and then gradually recovers, so a weak echo cannot re-trigger it, while a strong new input still can. We propose to give a transcription model a learned refractory period that spreads across pitches and instruments. After the model emits a note, the threshold for that note's harmonic relatives and for the same pitch on other instruments is raised, by an amount proportional to the note's strength, and it recovers on a learned time course.

In two sentences: a note's overtones are the main source of false notes in dense mixtures, so after each detected note the model should briefly demand more evidence before it accepts another note at the pitches and tracks those overtones contaminate. We learn the shape of that "demand more evidence" kernel from data using the point-process mathematics neuroscientists developed for refractory neurons, and we test whether it removes ghost notes and instrument leakage without suppressing genuine repeated or doubled notes.

Core Claim

The claim is narrower than "add spiking neurons to transcription," and that narrowness is deliberate. Same-pitch refractoriness (a note cannot instantly re-trigger itself) is close to what onset-conditioned frame models, minimum inter-onset-interval post-processing, and envelope HMMs already provide. We claim that the useful and missing structure is lateral in pitch and instrument. The evidence that should suppress a candidate note is the recent emission of a note whose harmonics overlap it. This structure cannot be expressed by per-pitch HMMs such as the ADSR model of Deep Polyphonic ADSR Piano Note Transcription or the pitch-wise HMMs of An Efficient Temporally-Constrained Probabilistic Model for Multiple-Instrument Music Transcription. Both treat each pitch's lifecycle independently. The approach also targets, more directly than added memory does, the cross-track duplication that MR-MT3 measures as instrument leakage.

Concretely, a backbone network produces a non-negative "free drive" $q_{p,i}(t)$ for an onset at pitch $p$ on instrument $i$ at frame $t$. The emitted onset intensity is

$$\lambda_{p,i}(t) = q_{p,i}(t)\,\exp\Big(-\sum_{k \in H_t} a_k\, R(t-t_k)\, P(p-p_k)\, S(i,i_k)\Big),$$

Here $H_t$ is the set of previously emitted onsets $k$, each with time $t_k$, pitch $p_k$, instrument $i_k$, and predicted amplitude $a_k$. The function $R$ is a recovery curve over elapsed time, parameterized as a small sum of decaying exponentials. $P$ is a learned weight over signed pitch intervals in the range ±36 semitones. $S$ is a learned instrument-pair weight. The product form, free rate times recovery function, is taken directly from Refractoriness and Neural Precision. That work used it to explain why retinal ganglion cells fire far more regularly than a Poisson model predicts, reproducing a Fano factor of 0.266 against a biological 0.250, where Poisson gives 1.0.

We generalize that recovery function from one neuron's own history to a history of other sources. The kernel is multiplicative and finite, so the threshold is raised but never clamped. This follows the lesson of Incorporating the Refractory Period into Spiking Neural Networks through Spike-Triggered Threshold Dynamics, which showed that hard clamping blinds neurons to input during the refractory window. Its RPLIF neuron instead raises the threshold multiplicatively while integrating continuously. The amplitude scaling $a_k$ means a loud note suppresses its overtone positions more than a quiet one. This loosely parallels the finding in Modelling the influence of short term depression in vesicle release that recovery in the auditory nerve depends on how much was recently "spent," not only on post-spike time.

Research Program

The backbone is a frame-level multi-instrument transcriber of the Onsets-and-Frames family, in the form used as the transcription module of Jointist. It outputs per-pitch, per-instrument onset, frame, and velocity predictions. We choose a frame model rather than a token decoder like MT3 because the refractory readout needs an explicit intensity per pitch, instrument, and frame. The kernel adds a few hundred parameters: about 73 pitch-interval weights, $34^2$ instrument-pair weights at most (factorized to low rank), and a handful of recovery time constants. Any difference from the backbone therefore cannot be attributed to capacity.

Training sidesteps the serial bottleneck that SPikE-SSM attacks with FFT-based boundary compression. Because $H_t$ in training can be the ground-truth onset history, the exponent becomes a fixed convolution of the label tensor with the kernel. This is computable in parallel, and the whole model trains by maximizing a discretized point-process log-likelihood, a Bernoulli loss per frame with $\lambda$ as the rate. Feeding true history at training time and predicted history at test time creates exposure bias. We handle it with scheduled replacement of ground-truth history by the model's own thresholded emissions, and we report results with and without it. Inference is a cheap causal scan over frames, since the kernel's support is short.

The analysis side treats the learned kernel as a measurement. If the hypothesis is right, $P$ should place clear suppressive mass at +12, +19, and +24 semitones, and asymmetrically less at negative intervals, because overtones lie above the fundamental. $S$ should be large for instrument pairs that the backbone confuses. $R$ should recover on the time scale of note attacks, tens of milliseconds, not the millisecond scale of neurons. A kernel that learns only a same-pitch spike at interval 0 would falsify the cross-pitch claim, even if accuracy improved.

Baselines

The naive reference is the backbone with its standard fixed-threshold peak picking and no history dependence. Every gain is reported relative to this, on the same checkpoint where possible.

The strongest cheap competitor is tuned post-processing. It combines a per-pitch minimum inter-onset interval with hysteresis thresholding, where a higher threshold applies for a fixed window after an onset. This is a hand-set, same-pitch refractory period. We grid-search its window and threshold on the validation split with the same number of trials used to tune the kernel's regularization. Beating it requires a statistically significant onset-F1 gain on held-out Slakh2100 and on the multi-instrument portion of the 2025 AMT Challenge set.

The structured-temporal competitor is an ADSR-style per-pitch HMM decoded with Viterbi on the same backbone outputs, following Deep Polyphonic ADSR Piano Note Transcription. That paper reports 56.08% complete-note F-measure on the MAPS Disklavier subset against 50.22% for a BLSTM. On MAPS, matching or exceeding that HMM decoder applied to our own backbone is the bar. External absolute numbers are context only, because the backbones differ.

The strongest existing systems are memory and capacity approaches. MR-MT3 reports reducing the instrument leakage ratio on Slakh2100 from 1.70 (MT3) to 1.12, where 1.0 is ground truth, and raising instrument-detection F1 from roughly 34% to 41%. On the 2025 AMT Challenge, Advancing Multi-Instrument Music Transcription reports overall F1 of 0.3932 for MT3 and 0.5998 for the top system MIROS, with MIROS falling from 0.7193 on solo excerpts to 0.4585 on multi-instrument ones. For Jointist, the file reports only a gain of over 1 percentage point in transcription, so no absolute transcription F1 for Jointist was found on Lacuna. For YourMT3+, the file gives no exact Slakh onset-F1 table values, so no reported result was found on Lacuna for its Slakh onset F1. We would rerun MT3 and MR-MT3 from released configurations on our splits where feasible. The target is a leakage ratio at or below 1.12 with a frame backbone of comparable size.

The key ablations isolate the ingredient. The first restricts $P$ to interval 0 and $S$ to the identity, which gives same-pitch, same-instrument refractoriness, essentially a learned version of the post-processing baseline. The second drops $S$, making the kernel pitch-lateral only. The third drops the amplitude scaling $a_k$. The fourth replaces the multiplicative relative threshold with a hard clamp. The proposal stands only if the full kernel beats the same-pitch-only ablation.

Why not just…?

Why not just use minimum inter-onset intervals and hysteresis in post-processing? This is the real threat for same-pitch re-triggering, and we expect it to capture most of that effect. That is why the same-pitch-only ablation exists. Post-processing cannot suppress an octave ghost or a duplicate note on a second instrument without also suppressing legitimate octave doublings, which are common in orchestration. Fixing this requires a weighted, amplitude-dependent, cross-pitch rule, and that rule is the proposal. If the full kernel does not beat tuned post-processing on the harmonic-ghost metric defined below, the idea fails.

Why not just decode with an ADSR or sound-state HMM? The HMMs in Deep Polyphonic ADSR Piano Note Transcription and An Efficient Temporally-Constrained Probabilistic Model for Multiple-Instrument Music Transcription enforce lifecycle order within each pitch and are handcrafted. Coupling all pitches in an HMM blows up the state space, and the second paper reports instrument assignment as a persistent failure. Our kernel is a factorized coupling that HMMs cannot afford. The HMM baseline on an identical backbone will show whether cross-pitch coupling adds anything beyond lifecycle order. The two are also composable, and we will test the combination.

Why not just add memory or capacity, as MR-MT3 and YourMT3+ do? Attention can in principle learn harmonic suppression. Yet leakage remains above ground truth after memory is added (MR-MT3), large models hallucinate secondary instruments on solo audio, and every system in the 2025 challenge drops sharply on mixtures. That suggests the structure is not reliably learned from the data available. The fair test is the leakage ratio and the solo-recording hallucination rate. If an unconstrained model matches our kernel on both at equal size, the kernel's remaining value is interpretability and data efficiency, which we probe by training on 10% and 25% of Slakh.

Proposed Experiments

We train on Slakh2100-redux (Slakh2100). We evaluate on its test split, on the 2025 AMT Challenge set, and on URMP for real ensemble recordings. Separately, we train and evaluate on MAESTRO and MAPS to check that the method does not harm solo piano. We report mir_eval onset F1 and onset+offset F1, instrument-detection F1, the MR-MT3 leakage ratio, and two diagnostic metrics we define. The first is harmonic ghost rate: false-positive onsets within 100 ms after a true onset at intervals of +12, +19, or +24 semitones, or at the same pitch on another instrument. The second is genuine-repeat recall: recall restricted to true onsets that occur within the recovery window of a related true onset, such as repeated notes, octave doublings, and unisons.

The decisive check is the joint behaviour of these two numbers. The full kernel must cut the harmonic ghost rate substantially relative to both the naive backbone and the same-pitch ablation, while keeping genuine-repeat recall within a small margin of the backbone. A method that removes ghosts by also removing real doublings has learned a blunt filter, not refractoriness.

A second check is the solo-audio "complexity paradox." On the solo excerpts of the challenge set, the learned instrument kernel should reduce hallucinated secondary instruments relative to the backbone. A third check is the learned kernel's shape: overtone-interval peaks in $P$, and $S$ aligned with the backbone's confusion matrix. A fourth check is the exposure-bias gap between teacher-forced and self-generated history, which shows whether the parallel training trick costs accuracy at inference.

Phase cancellation is outside the scope. A refractory readout acts on event timing and cannot restore energy destroyed within a frequency bin.