Rhythm Correlation Functions in a Tempo-Aligned Time Frame

Speech rhythm is the pattern of strong and weak beats in how people talk. It matters for telling languages apart, for spotting motor speech disorders and cognitive decline, and for identifying speakers. The field still measures it with crude statistics, such as the spread of vowel durations or the average difference between neighboring intervals. Those numbers depend on hand-marked segment boundaries and are computed in raw seconds, so a speaker who speeds up and slows down within a sentence blurs whatever rhythmic structure is there. Particle physicists faced the same blurring when they studied electron-positron collisions. The jets in each collision point in a random direction, so correlations measured against the fixed beam axis average out to nothing. Their fix was to rotate each event into its own frame, the "thrust axis" along which most of the momentum flows. They then measured the joint distribution of particle pairs in that frame and divided out the pairs formed by mixing different events. That turns a blurred cloud into separable peaks and ridges.

We propose to do the same for speech. A dynamic-programming tracker (a global search that trades how well each moment fits the audio against a penalty for abrupt changes) follows the local syllable tempo and phase of each utterance directly from the audio envelope. It re-expresses time in "cycles elapsed" rather than seconds. In that utterance-specific frame we compute a two-particle correlation function over pairs of acoustic prominence events: how likely two syllable-nucleus events are at a given lag in cycles and relative phase, relative to a mixed-utterance background. The claim is that this object separates within-syllable coupling (a near-lag peak) from foot-level periodicity (ridges at lags of two to three cycles). It should also be less sensitive to tempo drift than interval metrics or envelope statistics computed in seconds. We test that claim with an ablation that removes the alignment and keeps everything else fixed.

What exists and what is missing

Three strands of the literature come close. The Generalized Variability Index, from Long-distance rhythmic dependencies and their application to automatic language identification, extends pairwise duration differences to lags up to nine intervals. It raises 11-way language identification from 67.05% with standard metrics to 79.05%. This paper motivates the proposal, since it shows long-range structure carries information, and it is our strongest competitor. It still works from transcribed intervals in absolute time, and it reduces each lag to a single scalar difference rather than a distribution over lag and phase. The phase-locking study, Instantaneous changes in acoustic signals reflect syllable progression and cross-linguistic syllable variation, maps each syllable to a phase angle from 0 to 2π. It shows that acoustic modulation locks to that phase: a phase-locking value near 0.67 in English against 0.56 for surrogate boundaries, and near 0.77 for Tashlhiyt vocalic nuclei. That supports a phase-based frame, but it uses manually annotated syllables and measures single-event coupling, not pair correlations. Rhythm Features for Speaker Identification provides the clearest evidence of blurring. Its rhythm-only identification drops from 39.01% on read LibriSpeech to 3.26% on conversational VoxCeleb1, using ASR alignments in fixed 20 ms frames.

The method comes from the dynamic-programming line. Visual Rhythm and Beat selects beats from a noisy impact envelope by trading impact strength against tempo consistency, but it evaluates only qualitatively and never on speech. A Time-Warping Pitch Tracking Algorithm Considering Fast f0 Changes shows that searching over candidate rates of change inside a Viterbi decoder handles fast, non-stationary trajectories in babble noise. Spoken Language Modeling with Duration-Penalized Self-Supervised Units shows that a single transition penalty can impose duration structure on unlabeled speech. Its flat penalty has no notion of periodicity, and the authors report a cost in phonetic discrimination. We reuse the local-plus-transition cost form from these papers. What we add is a state space of tempo and phase, so the decoder outputs a warped time axis rather than a segmentation.

The physics papers supply the analysis design and one warning. In Two-particle angular correlations in e+e− collisions to hadronic final states in two reference coordinates at Belle, switching to the thrust frame resolves structure that the beam frame blurs. The ALEPH archival analyses, such as Long-range angular correlations of charged particles in high multiplicity e+e− collisions at 91 GeV using archived data from the ALEPH detector at LEP, use the same event-intrinsic frame together with mixed-event backgrounds. The same Belle paper also found an apparent enhancement in on-resonance data. Toy simulations traced it to the thrust axis lying in the decay plane of the two B mesons, which is a kinematic artifact of the alignment itself. This constrains our design. A tracker built to find periodicity can manufacture periodicity, so artifact checks with synthetic data are a required part of the program.

Core Claim

Rhythm is better described as a pair distribution in a tempo-aligned frame than as a set of interval statistics in seconds. Two consequences follow. First, the alignment should carry measurable information beyond a fixed or globally rescaled time axis, especially in spontaneous and pathological speech where tempo drifts. Second, the lag-and-phase structure of the correlation should give stress-timed and syllable-timed languages different ridge patterns at foot-level lags, even where metrics computed over syllables and feet cluster languages counter-intuitively (Rhythm metrics on syllables and feet do not work as expected).

Research Program

The first component is event detection. We compute a band-limited amplitude envelope and an MFCC spectral-modulation track, following the features of the phase-locking study. Candidate syllable-nucleus events are local envelope peaks, each weighted by prominence (peak height relative to neighbors). Detection is kept deliberately permissive, because the tracker, not the detector, decides which candidates land on beats.

The second component is the tempo-phase tracker. At each frame $t$ the hidden state is a local period $\tau_t$, from a grid of about 30 values spanning roughly 3 to 9 syllables per second, and a phase $\theta_t$. The tracker minimizes

$$\sum_t \Big[-\log E(t)\,\mathbb{1}[\theta_t \approx 0] \;+\; \lambda\,\big(\log\tau_t-\log\tau_{t-1}\big)^2 \;+\; \mu\,\mathbb{1}[\tau_t = 2^{\pm1}\tau_{t-1}]\Big],$$

subject to the phase advancing by $2\pi/\tau_t$ per frame. Here $E(t)$ is the event strength at frame $t$, $\lambda$ penalizes gradual tempo change, and $\mu$ is a separate cost for halving or doubling the period. The last term plays the role that octave-jump penalties play in pitch trackers. The output is the unwrapped phase $u(t)=\sum_{s\le t}1/\tau_s$, which gives time in cycles.

$\lambda$ and $\mu$ are chosen once and then frozen. They are set to maximize agreement between tracked phase and the manually annotated syllable phase on the English data of the phase-locking study. That keeps the known sensitivity of dynamic-programming trade-offs away from any downstream label.

The third component is the correlation function. Each event $i$ in an utterance gets a cycle coordinate $u_i$ and a prominence class $c_i$ (strong or weak, split at the utterance median). For a lag $\Delta u = u_j-u_i$, split into an integer cycle count $\Delta n$ and a fractional phase $\Delta\theta$, we define

$$C_{ab}(\Delta n,\Delta\theta)=\frac{S_{ab}(\Delta n,\Delta\theta)}{B_{ab}(\Delta n,\Delta\theta)}.$$

$S_{ab}$ is the prominence-weighted count of same-utterance pairs of classes $a,b$, and $B_{ab}$ is the same count over pairs whose two members come from different utterances with matched event count and cycle length. This is the mixed-event normalization from the collider analyses. It removes structure that comes only from utterance length and event density. The feature vector is $C$ sampled over $\Delta n\le 8$ (matching the GVI range) and 8 phase bins, for three class pairs.

Baselines

The naive reference point is the standard rhythm-metric set of %V, ΔC, ΔV and nPVI. On the Globalphone-plus-TIMIT 11-way language identification task, it reaches 67.05% in Long-distance rhythmic dependencies and their application to automatic language identification. The strongest existing method is that paper's baseline-plus-GVI combination at lags up to nine, which reports 79.05%. It is the right competitor because it already exploits long-range lags, which is our main selling point. We follow its protocol and language set. If the exact splits cannot be recovered, we reimplement GVI on our splits, report both numbers, and base comparisons only on our reimplementation. All feature sets feed the same classifier family, with the same hyperparameter search budget. Beating it means exceeding GVI plus baseline on held-out accuracy, with the difference outside the bootstrap confidence interval. We also require that our features alone approach GVI alone at lag 8 (67.17%) without using transcripts, since GVI relies on transcribed intervals.

The most dangerous simple baseline is the autocorrelation or modulation spectrum of the amplitude envelope, computed in absolute seconds. It needs no events and no tracker. We found no reported result on Lacuna for envelope modulation features on these tasks, so we run it ourselves at matched dimensionality.

Two ablations isolate the key ingredient. The first computes the identical correlation function $C_{ab}$ in raw seconds, with no tracker. The second computes it after a single global per-utterance rate rescaling, which is the linear normalization a skeptic would propose. If the dynamic-programming frame does not beat both ablations, the alignment claim fails regardless of any gain over GVI. A third ablation replaces the mixed-event denominator with a flat one, which tests whether the physics-style background subtraction matters.

For the clinical tasks, the references are the reported dysarthria results in Dysarthria Detection and Severity Assessment Using Rhythm-Based Metrics: 85.7% MLP detection on TORGO with baseline plus rhythm features, 97.8% random-forest detection on QoLT, and 63.6% SVM severity classification on QoLT. We substitute our features for their rhythm block, keep their classifiers, and use speaker-independent splits. For speaker identification, the reference is the rhythm-only 39.01% on LibriSpeech and 3.26% on VoxCeleb1 from Rhythm Features for Speaker Identification. The target is a smaller read-to-spontaneous ratio, not absolute accuracy competitive with acoustic embeddings.

Why not just…?

Why not just use GVI, which already looks at long lags? GVI is the main threat, and we treat it as the bar to beat rather than something to argue around. It collapses each lag to one scalar, cannot tell a strong-weak alternation from a weak-strong one, and inherits whatever tempo drift sits in the transcribed durations. If our features beat baseline metrics but not GVI, the honest conclusion is that the pair-distribution form adds nothing on language identification. We would report that and restrict the claim to the transcript-free and spontaneous-speech settings.

Why not just take the autocorrelation of the amplitude envelope? This baseline has most of our information content without the machinery, and it could plausibly match us on read speech, where tempo is steady. Our prediction is specific: the gap should open on conversational and pathological speech, where the rate changes within an utterance. The seconds-frame ablation of our own correlation function measures exactly that gap. If it does not open on VoxCeleb1 or TORGO, the alignment is not doing its job.

Why not just normalize by global speaking rate? Category-relative scaling, as in Utterance-level normalization for relative articulation rate analysis, needs phone labels. A global rescaling needs no labels but corrects only for the average rate, not for acceleration within the utterance. The global-rescaling ablation tests this directly. If it matches the tracked frame, we drop the tracker.

Why not just pool WavLM features or use a forced aligner? In the speaker-identification study, fusing rhythm with WavLM and x-vector embeddings gave no significant gain, and WhisperX-derived rhythm collapsed on spontaneous speech. Aligners also need transcripts, which do not exist for unwritten languages or severely dysarthric speech. We include a pooled-WavLM language-identification classifier as a reference, but a win against it is not our claim. Our goal is an interpretable rhythm measure that is robust to tempo, not the best possible classifier.

Decisive Checks

The first check tests the tracker against known structure. On the English and Tashlhiyt data of the phase-locking study, the acoustic features should lock to tracked phase at least as well as to manual syllable phase (joint phase-locking near 0.67 in English), and clearly above the surrogate level near 0.56. Nucleus detection on CLIPS is scored with the same precision, recall and F1 procedure that gave Speechmark an F1 of 0.415 in Automatic Detection of Rhythmic Features in Pathological Speech of MCI and Dementia Patients.

The second check tests for artifacts, following the Belle B-meson lesson. We generate synthetic event trains from renewal processes with drifting rate and no foot structure. We also build surrogates from real speech by shuffling its intervals. Both go through the full pipeline. Any ridge at $\Delta n\ge2$ that survives in these null data is a tracker artifact, and we report the correlation function only after subtracting the null. This check must pass before any linguistic claim is made.

The third check is the typology test. On the North Wind and Sun recordings used in Rhythm metrics on syllables and feet do not work as expected, English and German should show a strong-to-strong ridge at $\Delta n$ of 2 to 3 that syllable-timed languages lack. If German again clusters with the syllable-timed languages, that is a substantive finding against the stress-timing account, not a failure of the method.

The fourth check targets the clinical case. On the Italian MCI and dementia corpus, where standard metrics showed no group differences, the same group comparison run on our features separates two explanations for that null result: segmentation error, or adjacent-lag myopia. This check depends on getting access to that corpus.

The language-identification comparison against GVI and the seconds-frame ablation together decide whether the proposal holds up.