Unmixing the Rhythm Before Choosing the Tempo

When a tempo estimator gets a song wrong, it rarely produces an arbitrary number. It reports twice or half the true tempo, or occasionally a third of it. The correct value is almost always among its candidates, but the estimator misjudges which candidate is the main pulse (Pikrakis & Theodoridis report this for empirical-mode-decomposition tempo induction). We propose that much of this "octave ambiguity" comes from mixing. The hi-hat ticks at twice the beat, the kick and bass sit on the beat, the chords change once per bar, and a fast solo fills in eighth or sixteenth notes. Summed into one onset curve, these streams produce a stack of peaks at related tempos with no indication of which instrument made which peak. If we first unmix the rhythmic surface into separate onset streams using blind source separation (recovering hidden signals from their mixtures without knowing how they were mixed), and then reason about the tempo ratios between streams, the main pulse becomes the level on which the streams agree rather than simply the loudest peak.

The proposal has two parts. The first is a cheap, instrument-agnostic unmixing step: independent component analysis (ICA, a method that finds a linear transform making output signals statistically independent) applied to onset envelopes from many frequency bands. The second is a small "ratio-consensus" network that takes one tempo distribution per stream on a logarithmic tempo axis. On that axis, doubling and tripling become fixed shifts, so a convolution can learn which cross-stream patterns, such as "stream A at 2×, streams B and C at 1×," point to the true pulse. Because the network learns ratios of 2 and 3 rather than having powers of two built in, it should handle waltzes and compound meters, where current binary-biased designs struggle.

Core Claim

Mix-level tempo models see an aggregate salience curve. Our claim is that the information needed to resolve octave errors lies in how salience at each metrical level is distributed across sources, and that blind unmixing of subband onset envelopes recovers enough of this distribution to beat mix-level models on strict tempo accuracy. Strict accuracy (called Accuracy 1, within 4% of the true tempo with no octave tolerance) is the right target. Lenient Accuracy 2 forgives octave errors, so the gap between the two measures exactly the failure we address.

The modeling assumption is concrete. Let $e_k(t)$ be the spectral-flux onset envelope in frequency band $k$, and let $o_i(t)$ be the onset train of rhythmic source $i$. Because each instrument occupies a characteristic set of bands, we approximate

$$e(t) \approx A\, o(t),$$

where $A$ holds each source's spectral footprint. This is the instantaneous linear mixture that classical ICA addresses. Onset trains are sparse and spiky, which means strongly non-Gaussian, and that is the regime where the natural-gradient ICA of Amari et al. works well. Its equivariance property (convergence does not depend on how ill-conditioned $A$ is) matters here because neighbouring bands are highly correlated.

Relation to Prior Work

The obvious version of this intersection, "separate stems, then track rhythm," already exists, and we must differ from it clearly. Beat Transformer runs a frozen Spleeter model to obtain five fixed stems and alternates attention over time with attention across stems. It gains 4 F-measure points in downbeat tracking on unseen GTZAN, which motivates our hypothesis that per-source views carry metrical-level information. Source Separation-based Data Augmentation uses separated stems only to create training variety. Alonso, Richard & David split audio into harmonic and noise subspaces with a Hankel eigendecomposition, which costs about 80% of their compute. They reach about 92% within 5% tolerance on 961 excerpts.

All three treat separation as a fixed front end tied either to a pop-centric stem vocabulary or to a two-way tonal/transient split. None models tempo ratios between sources explicitly. Our unmixing needs no instrument labels, so it applies to tabla and sitar or Afro-Latin percussion ensembles where Spleeter's vocabulary does not fit. It is also cheap (ICA on roughly 20 to 40 envelope channels), and its output is consumed by a component whose sole job is ratio reasoning.

On the tempo side, Böck & Davies add parallel 2:1 and 4:1 dilations to a temporal convolutional network and obtain over 83% Accuracy 1 on three unseen datasets. Böck, Davies & Knees show that a 29,901-parameter multitask network reaches over 93.8% Accuracy 2. Gagneré, Peeters & Essid build $2^n$ positive pairs into a contrastive objective. These papers supply our per-stream estimator, since we reuse the compact multitask design. They also constrain the design: their binary assumptions are the reason we learn ratios of 2 and 3 instead of fixing them. Chowdhury et al. already exploit synchrony across sub-bands. Their work is the closest non-separation competitor, and our key ablation asks whether unmixing adds anything beyond raw sub-band views. Xiao et al. find that timbre predicts perceived tempo, which suggests that knowing which spectral footprint carries a pulse should help. The ICA mixing columns provide exactly that footprint.

One assumption runs directly against the separation literature. ICA assumes independent sources, but musicians deliberately lock to a shared grid. We expect this to hurt less than it seems, for three reasons. Separation here is driven by spectral footprints and spike timing within the grid rather than by grid alignment. Shared periodic statistics are exactly the cyclostationary structure that single-channel cyclostationary separation shows can help separation rather than hurt it. If ICA still fails, the ratio-consensus network accepts neural stems unchanged. We test this assumption directly instead of assuming it holds.

Research Program

We will build three components. The first computes about 32 log-spaced band onset envelopes at 100 frames per second and applies natural-gradient ICA in sliding 10-second windows to obtain $N$ streams. A fixed $N=6$ is the default, with dimension selection by explained variance as a variant. ICA leaves the order and scale of the outputs undetermined, as do all blind methods. We do not fix this with sorting rules of the kind used in constrained ICA. Instead, every later layer treats the streams as an unordered set, so their order does not matter. We normalize each stream to unit variance.

The second component is a shared compact multitask network, copied from Böck, Davies & Knees, which maps each stream (plus the full mix as an extra stream) to a tempo distribution on a log-tempo grid. The third is the ratio-consensus head. For each stream it stacks the tempo distribution and the stream's ICA mixing column, which describes the stream's spectral footprint. It applies a log-tempo convolution with a kernel wide enough to span ratios of 1/3 to 3, pools symmetrically across streams so that stream order cannot affect the output, and produces a single tempo distribution trained with cross-entropy against the annotated tempo. The full system is trained end to end, except for ICA, which has no learned parameters.

We will also analyse what the head learns. We will plot learned kernel weights at shifts of log 2 and log 3, test whether ICA streams line up with known stems on tracks where stems are available (by measuring correlation with Spleeter-stem envelopes), and break down Accuracy 1 errors by meter and genre.

Baselines

The naive reference picks the highest autocorrelation peak of the summed onset envelope within 30 to 300 BPM. It shows how severe the octave problem is without learning. A second naive point is a genre-typical tempo prior applied to that peak hierarchy, because on Ballroom a prior alone can resolve many octaves.

The strongest existing method is the dilated multitask network of Böck & Davies, which reports over 83% Accuracy 1 across three unseen benchmark datasets. We will retrain it on the same training split with the same hop-size augmentation and an equal hyperparameter-search budget, measured in GPU hours. Beating it means a higher Accuracy 1 on held-out datasets, with a paired bootstrap over tracks giving a confidence interval that excludes zero. We expect the largest margins on HJDB, the Ballroom waltz classes, and the EDM corpora of Knees et al..

The second strong competitor is Beat Transformer, with tempo read from the median inter-beat interval of its output. No reported tempo Accuracy 1 result was found on Lacuna for Beat Transformer, so we will run it ourselves with its released settings.

Three ablations isolate the key ingredient. In the first, the consensus head receives raw band envelopes grouped into $N$ fixed bands with no unmixing. This is a sub-band synchrony variant in the spirit of Chowdhury et al.. In the second, the head receives Spleeter stems instead of ICA streams. In the third, the head is limited to power-of-two shifts. Parameter counts are matched across all variants.

Why not just…?

Why not just train a bigger mix-level tempo classifier with octave-aware losses? This is the most serious threat, because a deep network applied to the spectrogram can in principle learn the unmixing internally. We address it with a matched-capacity comparison. We will scale the Böck–Davies network up to our total parameter count and give it the same data. If its Accuracy 1 matches ours on every held-out set, our claim fails. We would then report that explicit unmixing is unnecessary and that the result holds only for very compact models, where the 29,901-parameter regime may favour structural help.

Why not just use sub-band views without separation? Chowdhury et al. already use synchrony across bands. Bands mix instruments, however: a kick drum and a bass guitar share low bands, and a hi-hat spreads over many high bands. The fixed-band ablation answers this directly. If it matches ICA, the separation contribution is nil.

Why not just use Beat Transformer's stems? Its stems were designed for pop instrumentation and are frozen. On sitar and sarod concerts, where fast solos inflate event density (Vinutha et al.), and on Afro-Latin ensembles (Fuentes et al.), several instruments may fall into "other." The Spleeter-input ablation and per-corpus results will show whether blind streams generalize better or whether pretrained stems are simply better everywhere. Either outcome is informative.

Why not just apply a tempo prior? Priors fix typical-tempo genres but fail when a genre spans octaves. Drum-and-bass at 170 BPM is often felt at 85, for example. The prior-only baseline quantifies how much of any gain is just prior knowledge.

Proposed Experiments

We will train on the standard tempo-annotated pool and use leave-one-dataset-out testing over Ballroom, GTZAN, HJDB, the two EDM tempo sets of Knees et al., and, as stress tests outside the binary-meter assumption, the sitar/sarod and Afro-Latin collections. ASAP piano will serve as an expressive-timing case where we expect little gain, because a solo piano offers little to unmix. That makes it a useful negative control. The metrics are Accuracy 1, Accuracy 2, and their gap, with the gap also broken down by error type (×2, ×1/2, ×3, ×1/3).

Two checks would decide the question. In the first, ICA streams beat fixed bands at matched capacity on Accuracy 1, with the gain concentrated in reduced ×2 and ×1/2 errors on HJDB and EDM. In the second, removing ratio-3 shifts from the head selectively hurts the waltz classes. A sanity experiment checks the independence assumption directly. We will build mixtures from the onset envelopes of separately available stems, run ICA, and measure how well the recovered streams correlate with the true stem envelopes. If recovery is poor, we switch to neural stems and report that blind unmixing of rhythm does not hold up under grid-locked sources.