Many systems that find events in signals, such as note onsets in piano audio, sleep spindles in brain recordings, or goals in soccer video, now split the job in two. A classifier says whether each short frame of the signal contains an event. A regressor says how far the frame's center is from the event's exact start or end, which gives precision finer than one frame. These two outputs are produced independently, frame by frame, and are then stitched into events by hand-written rules. This note proposes replacing those rules with a learned, exactly solvable probabilistic decoder. The decoder treats every frame's distance prediction as a noisy vote for one shared, continuous boundary time. It decides which frames vote for the same boundary, fuses their votes into a single estimate with an error bar, and enforces legal event structure (start, then sustain, then end) in the same step.
The opportunity is concrete. The best regression-based piano transcriber turns frame-wise distance predictions into notes with an analytical peak-picking procedure that has no learned parameters (High-Resolution Piano Transcription With Pedals by Regressing Onset and Offset Times). Its classification head has no sequence constraints, so it can produce broken notes and inconsistent pedal durations. Conditional random fields (CRFs) fix exactly this kind of fragmentation in text labeling. A CRF is a model that scores an entire label sequence jointly rather than one position at a time. Text CRFs, however, only handle discrete labels and have no way to represent a continuous time offset.
Core Claim
The obvious bridge between the two lines of work is to put a standard linear-chain CRF on the classification head and leave the regression head alone. That is probably a weak idea. The backbones already carry bidirectional recurrent context, and a CRF over frame labels does nothing to resolve where inside a frame the boundary lies. Our claim is different. The continuous offsets themselves should define the potentials of the structured model.
Consider a run of consecutive frames near a boundary. Each frame predicts a Gaussian over the absolute boundary time: a mean $m_t$ (frame center plus the regressed offset) and a variance $\sigma_t^2$ from a second output. If all of these frames are scoring the same boundary $\tau$, then integrating $\tau$ out of the product of their Gaussians gives a closed-form agreement score. That score depends only on running sums of $1/\sigma_t^2$, $m_t/\sigma_t^2$ and $m_t^2/\sigma_t^2$. It can therefore be extended frame by frame in constant time. This turns the problem into a semi-Markov CRF, a CRF whose units are variable-length segments rather than single frames. In it, "boundary-voting regions" are segments whose score is this fused Gaussian evidence, and the remaining segments carry discrete phase labels such as background, active and pedal-down. Exact Viterbi and forward–backward inference cost $O(T \cdot L \cdot |Y|^2)$, where $T$ is the number of frames, $L$ is a small maximum voting-region length (a few frames), and $|Y|$ is the number of phase labels.
The decoded output is a legal segmentation together with, for every boundary, a Gaussian posterior. Its mean is the precision-weighted average of the votes and its variance is $(\sum_t 1/\sigma_t^2)^{-1}$. Training maximizes the joint likelihood of the discrete structure and the observed continuous boundary times:
$$ \log p(y, \tau \mid x) = \log p(y \mid x) + \sum_{k} \log \mathcal{N}\big(\tau_k \mid \hat\mu_k(x, y), \hat v_k(x, y)\big), $$
Here $y$ is the gold segmentation, $\tau_k$ is the annotated time of the $k$-th boundary, and $\hat\mu_k, \hat v_k$ are the fused mean and variance of the frames assigned to that boundary under $y$. Gradients reach both heads through one objective. This replaces the separate, hand-weighted classification and regression losses used across the dual-head literature.
Relation to Prior Work
Three regression-based detectors motivate the gap. Kong et al. regress onset and offset distances with a convolutional-recurrent network and then decode analytically (High-Resolution Piano Transcription). RMS-Net pairs a classification head with an offset regression head on soccer snippets and explicitly assumes at most one event per snippet (RMS-Net). A sleep-EEG detector adapts one-dimensional SSD/YOLO-style detection with a multi-task loss (A Deep Learning Architecture to Detect Events in EEG Signals During Sleep). All three let each frame or anchor regress in isolation and reconcile the results with heuristics or suppression. We make the reconciliation learned, probabilistic and exact.
The structured side supplies the method. Jagannatha and Yu show that context-dependent pairwise potentials and approximate long-range message passing reduce entity fragmentation in clinical text (Structured prediction models for RNN based sequence labeling in clinical text). Their potentials condition on text context. Ours condition on continuous time-to-boundary predictions, which no prior model does.
Structured Maximal Sums is the closest temporal precedent (Temporal Action Localization by Structured Maximal Sums). It decomposes actions into start, middle and end phases and finds windows by exact dynamic programming. Its boundaries snap to the frame grid, and its scores are unnormalized sums with no uncertainty attached. Our model keeps the phase grammar but makes the boundary a continuous latent variable with a posterior.
Soomro et al. use a CRF for temporal smoothing of superpixel segmentation in online localization (Predicting the Where and What of Actors and Actions through Online Action Localization). That is a discrete, pipeline-stage use, not a decoder for regressed times.
Bregman CRFs show that CRF inference can be made parallel on GPUs (Bregman Conditional Random Fields). This is relevant only as a later engineering path for very long streams. It is not part of the core claim, because our short-$L$ semi-Markov recursion should already be cheap at audio frame rates.
Research Program
We will build the decoder on top of the Kong et al. backbone, so that the only change is the output layer and the objective. The regression head gains a log-variance output per boundary type. The phase grammar for notes is background, onset-voting region, active, offset-voting region, then background again. For the sustain pedal it is the same with pedal-down replacing active.
Three analyses accompany the build. The first measures the error of fused boundary times as a function of how many frames vote. If the frame errors are roughly independent, that error should shrink with the number of voters, as an average of independent noisy estimates does. If frame errors are highly correlated, fusion buys little, and we will report this. The second checks calibration: whether the posterior variances predict actual timing errors, using MAESTRO's roughly 3 ms audio–MIDI alignment as ground truth (Enabling Factorized Piano Music Modeling and Generation with the MAESTRO Dataset). The third tests generality by applying the same decoder to spindle and K-complex detection on MASS SS2. Those events are longer, and annotators disagree about their boundaries.
Baselines
The naive reference point is the same backbone with frame-wise thresholding. An onset is declared wherever the onset probability peaks above a threshold, with no regression. This shows how much the continuous offsets contribute at all.
The strongest existing method is Kong et al.'s regression model with its analytical decoder. It reports note F1 of 96.72% on MAESTRO, against 94.80% for the earlier onsets-and-frames system. It reports sustain-pedal event F1 of 91.86%. Under 50 ms label shifts its F1 falls only from 96.72% to 96.39%, while the frame-classification baseline falls from 93.92% to 76.52% (High-Resolution Piano Transcription). We will retrain it ourselves with identical data splits, hop size, backbone, training steps and threshold-tuning budget on the validation set. Our decoder will be compared with that retrained number, not with the published figure.
Two ablations isolate the key ingredient, which is offset-dependent potentials with evidence fusion. The first adds a plain linear-chain CRF on the classification head while keeping analytical offset decoding; this is the obvious first-generation idea. The second uses our semi-Markov structure but takes each boundary time from the single most confident frame instead of the fused posterior.
On MASS SS2 the file reports only that the event detector led eight spindle baselines at an intersection-over-union (IoU) threshold of 0.3. No reported numeric result was found on Lacuna for MASS SS2. We will therefore reimplement that one-dimensional detector as the baseline.
Beating Kong et al. means one of two outcomes at matched compute. The first is a statistically reliable gain in onset-and-offset note F1 or in pedal F1, measured across multiple training seeds. The second is equal F1 together with clearly lower boundary timing error at tolerances tighter than the standard 50 ms. The pedal is where we expect the largest gain, because long pedal states are most exposed to fragmentation.
Why not just…?
Why not just keep Kong et al.'s analytical decoder and train a bigger backbone? This is the main threat on note onsets, where 96.72% F1 leaves little headroom. The analytical decoder already uses neighboring frames to locate a boundary. Our argument is narrower. The heuristic cannot weight frames by confidence, cannot enforce legal pedal durations, and gives no uncertainty. If the matched comparison shows no gain on pedals, on tight-tolerance timing or on coarse hop sizes, the honest conclusion is that the heuristic suffices.
Why not just add a linear-chain CRF to the classification head? This is our first ablation. Jagannatha and Yu show such layers reduce fragmentation in text (Structured prediction models). But a discrete CRF cannot see the offsets, so it cannot improve sub-frame precision. We expect it to help pedal fragmentation slightly and onset timing not at all. If it matches the full model, the continuous potentials are unnecessary.
Why not just raise the frame rate? The file notes that smaller hops multiply compute and worsen the imbalance between rare event frames and abundant background frames. The decisive check here is to train every system at 2× and 4× coarser hops. If evidence fusion is doing real work, our accuracy should degrade much more slowly than that of the analytical decoder or the frame classifier.
Why not just decode autoregressively? A sequence-to-sequence decoder learns the event grammar implicitly (Exploring Sequence-to-Sequence Learning in Aspect Term Extraction). It gives no exact posterior over boundary times and requires one sequential pass per output step. We will report decoding latency per second of audio alongside accuracy. We do not claim an accuracy advantage over such decoders without running one.
Proposed Experiments
The main experiment uses MAESTRO v2 with the standard splits and mir_eval note metrics, reported at the usual 50 ms onset tolerance and also at 10 ms and 20 ms. We also report median and 90th-percentile absolute onset error in milliseconds, and pedal event F1.
The robustness experiment repeats Kong et al.'s label-shift protocol at up to 50 ms. The fused variances should widen under shift rather than fail silently.
Calibration is measured by the fraction of true boundaries that fall inside the decoder's nominal 68% and 95% intervals. A useful uncertainty estimate must be close to nominal.
On MASS SS2 we report precision and recall at IoU 0.3 and above, against each expert annotator separately. Disagreement between the experts should correlate with the posterior variance.
We exclude SoccerNet as a primary benchmark. Its 5 to 60 second tolerance windows (SoccerNet) make sub-frame precision irrelevant. The phase grammar for multi-event soccer sequences is a natural extension, but it would test a different claim.
The idea fails if three things all hold at matched compute: no gain over the retrained Kong et al. decoder on pedal F1, no gain on tight-tolerance onset error, and no gain at coarse hop sizes.
References
- High-Resolution Piano Transcription With Pedals by Regressing Onset and Offset Times
- RMS-Net: Regression and Masking for Soccer Event Spotting
- A deep learning architecture to detect events in EEG signals during sleep
- Structured prediction models for RNN based sequence labeling in clinical text
- Temporal Action Localization by Structured Maximal Sums
- Predicting the Where and What of Actors and Actions through Online Action Localization
- Bregman Conditional Random Fields: Sequence Labeling with Parallelizable Inference Algorithms
- Enabling Factorized Piano Music Modeling and Generation with the MAESTRO Dataset
- Exploring Sequence-to-Sequence Learning in Aspect Term Extraction
- SoccerNet: A Scalable Dataset for Action Spotting in Soccer Videos