Fusing Only What Clock Drift Leaves Coherent

Microphones on different devices (a smart speaker, a laptop, a pair of earbuds) each run on their own cheap crystal oscillator, and no two of these tick at exactly the same rate. A mismatch of 50 to 100 parts per million sounds negligible. Over a meeting, though, it slides one recording against another by hundreds of milliseconds. Because a timing error becomes a phase error proportional to acoustic frequency, the high frequencies lose cross-device alignment first. Current systems respond in one of two blunt ways. Some resample once and then treat all devices as if they were synchronized. Others give up on cross-device phase entirely and share only per-device "is speech present here" masks. The second option is known to break exactly in the high-frequency range, where wearables pick up competing talkers.

We propose to make this choice per frequency, per device pair, and over time, using a quantity the system can predict: how much phase coherence the remaining clock uncertainty still allows. We borrow one idea from graph-based recommendation, where averaging over neighbors is known to erase the individual signal that matters most. Following that idea, the fusion step keeps each device's own deviation from the network consensus instead of averaging it away. The claim is that a model of how drift destroys coherence, combined with a fusion filter that does not over-smooth, beats both "resample and beamform" and "decouple and mask" on long recordings where drift varies over time.

Core Claim

Model the devices as nodes of a graph, separately for each short-time Fourier transform frequency bin $\omega$ and each time $t$. Let $\varepsilon_{ij}$ be the relative sampling-rate offset between devices $i$ and $j$. The phase error it causes grows as $\Delta\theta = \omega\,\varepsilon_{ij}\,t$, which is the linear phase drift model of Joint Optimization of Sampling Rate Offsets Based on Entire Signal Relationship Among Distributed Microphones. Suppose an estimator leaves a residual offset error with variance $\sigma_{ij}^2(t)$. This variance grows between re-estimations if the true offset wanders, for example with temperature. Under Gaussian error, the expected cross-device coherence that survives is

$$ w_{ij}(\omega,t) = \exp\!\left(-\tfrac{1}{2}\,\omega^2 t^2 \sigma_{ij}^2(t)\right), $$

which follows from the characteristic function of a Gaussian phase error. These weights define a frequency- and time-dependent graph with Laplacian $L(\omega,t)$, the matrix whose eigenvectors separate signal components that agree across devices (low graph frequency) from components that differ between devices (high graph frequency). The fused estimate is

$$ \hat{x}(\omega,t) = h\big(L(\omega,t)\big)\,y + g\big(L(\omega,t)\big)\,y, $$

where $y$ holds the per-device spectra and $h$ is a low-pass graph filter that performs coherent combination wherever the weights allow. The term $g$ is a learned high-pass branch that retains device-specific content, such as a wearer's near-field speech, rather than averaging it into the consensus.

The prediction is concrete and falsifiable. The acoustic band over which coherent fusion helps should shrink roughly as $1/(t\,\sigma)$ between resynchronizations, and the model should exploit that structure explicitly instead of learning it from short clips.

Relation to Prior Work

Three papers motivate the gap. SAMbA aligns asynchronous nodes with a temporal attention mechanism and survives start-time offsets of up to 64 ms. It concluded that rate offset was negligible, but only on 5–10 s clips, so cumulative drift was never tested. Cross-Talk Reduction absorbs jitter with look-ahead prediction taps and reports that its prediction filter degrades under significant drift. Cooperative Speech Separation With a Microphone Array and Asynchronous Wearable Devices zeroes all wearable-to-array cross-covariances to stay robust at ±50 ppm. It then finds that its masks fail at high frequencies because of cross-talk leakage. Our proposal sits between the last two: it keeps cross-device phase where drift permits and discards it only where drift forbids.

Two papers supply methods. Joint maximum-likelihood rate-offset estimation across all $M$ devices (Miyabe et al.) provides our offset estimates. We additionally need their uncertainty, which that work does not use downstream. The subband Kalman tracking of Clock Skew Robust Acoustic Echo Cancellation treats skew as a state that evolves over time. We adopt this to produce $\sigma_{ij}^2(t)$ under time-varying drift, a use for which that echo-cancellation work was not designed.

The graph side constrains the fusion design. Blurring-Sharpening Process Models for Collaborative Filtering and Frequency-aware Graph Signal Processing for Collaborative Filtering show that pure neighborhood smoothing destroys individual signal. They recover it with a sharpening step and a cascaded high-pass-then-low-pass filter, respectively. JGCF further finds that the extreme low and high bands are informative while a middle band is noise. The acoustic analogue is Enhanced Deep Speech Separation in Clustered Ad Hoc Distributed Microphone Environments, which finds that channel-averaging layers (Transform-Average-Concatenate) collapse on widely spread microphones and fixes this with hard coherence clustering. Our graph replaces hard clusters with physically predicted soft weights that change with frequency and time.

One caution about the transfer. None of the recommendation papers deal with physical time, and their finding about the "noisy middle" band need not carry over to device graphs. We therefore borrow only the structural lesson (keep a high-pass branch) and learn the band shape from audio data.

Research Program

The work proceeds in three stages. The first is a measurement study, run before any model is built. On simulated and real recordings, we will compute cross-device coherence as a function of acoustic frequency and time since resynchronization. We then test whether the Gaussian decay formula, fed with the estimator's own uncertainty, predicts that measured coherence. If it does not, for example because real drift is dominated by jumps or packet loss rather than smooth wander, the method's premise fails and we report it.

The second stage builds the system. A Kalman tracker over the joint offset estimates outputs offsets and variances. These feed the weight computation, followed by a polynomial graph filter (Chebyshev, following Kim et al.) with a low-pass and a high-pass branch. The fused per-device spectra then drive a mask estimator and a multichannel Wiener filter. The networks are trained on long simulated sessions with drift that wanders over time.

The third stage is analysis. Since the weights are interpretable, we can show which bands and device pairs were fused coherently at each moment. This exposes the trade-off the earlier wearable-plus-array system had to fix by hand.

Baselines

The naive reference performs no compensation. It runs a synchronous multichannel Wiener filter with a mask estimator directly on unsynchronized signals, as in the distributed system of DNN-Based Mask Estimation for Distributed Speech Enhancement. That system assumed zero drift and reported about 3 dB signal-to-interference gain over single-node processing under that assumption. We expect this gain to vanish as drift accumulates. Beating the naive reference is necessary but uninformative.

The strongest classical competitor resamples every device with the joint maximum-likelihood offset estimates of Miyabe et al., re-estimated on a sliding window, and then applies the same mask estimator and Wiener filter. It is the strongest because it uses all inter-device relations: the paper shows it reaches stable estimates in 5 s versus 10 s for pairwise methods. No reported result found on Lacuna for its separation quality (SI-SDR or STOI) after resampling. We will run it with the same window lengths, the same mask network, and the same tuning budget as our method.

The strongest neural competitors are SAMbA and CTRnet, retrained on our long-session data with equal parameter counts and training steps. No reported numeric STOI found on Lacuna for SAMbA on DISCO, since its results appear only graphically. On real data we anchor to CTRnet's reported 22.6% diarization-assigned word error rate on CHiME-7 DASR, against 26.6% for guided source separation and 27.8% unprocessed (Cross-Talk Reduction).

We also include the decoupled wearable baseline, which shares masks only and uses no cross-covariances (Kalyanasundaram et al.). Three ablations isolate the key ingredients. The first replaces the predicted weights with a fixed frequency cutoff tuned on validation data. The second removes the high-pass branch $g$. The third substitutes oracle offsets, which bounds what better estimation alone could give. Beating the baselines means the following: a statistically reliable SI-SDR gain over windowed joint-resampling on sessions of 5 minutes or longer with wandering drift; no loss on short clips; and a gain over the fixed-cutoff ablation concentrated above roughly 2 kHz.

Why not just…?

Why not just resample with a good joint offset estimate and beamform as usual? This is the real threat. If offsets are estimated accurately and drift is nearly constant, the residual $\sigma$ is tiny, every weight is close to one, and our method reduces to this baseline. The claim lives entirely in the regime of time-varying drift and finite estimation windows, where resampling with a stale estimate leaves high-frequency phase errors. The oracle-offset ablation and the drift-wander sweep settle the question. If windowed resampling matches us on 5–15 minute sessions with realistic wander, we concede that uncertainty-aware fusion is unnecessary.

Why not just fuse coherently below a fixed cutoff and share masks above it? This hand-tuned rule captures the physics coarsely, and if drift statistics were stationary it might suffice. It cannot adapt to time since resynchronization, to device pairs with different oscillator quality, or to temperature transients. The fixed-cutoff ablation, tuned on validation sessions and tested on sessions with different drift profiles, measures exactly this gap.

Why not let an end-to-end network learn alignment implicitly, like SAMbA or the SY4 echo canceller? These models are trained and evaluated on windows where cumulative drift stays inside their receptive field. Elucidating Clock-drift Using Real-world Audios reports a 2 dB echo-cancellation gain, but for playback-capture echo cancellation, not multi-device fusion. Our long-session evaluation tests whether these models degrade with elapsed time while ours does not. Giving them extra context length at matched compute removes the easy excuse.

Why not just use clustering plus channel averaging? Hard coherence clusters (Kim et al.) are computed once and are the same for all frequencies. Drift makes coherence depend on frequency and time, so a fixed cluster is wrong at high frequencies late in a session. The no-high-pass ablation also tests whether averaging, even within good clusters, erases the wearable's near-field signal.

Proposed Experiments

Simulation extends the DISCO setup used by SAMbA (four nodes, up to 100 ppm offset and 64 ms start offset) and the Pyroomacoustics plus LibriSpeech ad-hoc setup to sessions of 1–15 minutes. Drift is modeled as a constant offset plus a slow random walk, and a wearable-plus-array configuration at ±50 ppm mirrors the cooperative-separation study. We report SI-SDR, STOI, signal-to-interference ratio, and per-band SI-SDR, plotted against elapsed time. Real-data evaluation uses CHiME-7 DASR word error rate against the 22.6% CTRnet figure, and SMS-WSJ-FF-CT with injected drift, where the drift-free figure reported for CTRnet is 26.5 dB SI-SDR against 14.7 dB unprocessed.

The decisive checks are three. First, the measured coherence must match the predicted decay; this is checked before any training. Second, the advantage over windowed joint-resampling must grow with session length and drift wander and be near zero for constant drift. Third, the gain over the decoupled-mask baseline must be concentrated in exactly the high-frequency bands where that baseline is known to leak cross-talk. If the second check fails, the paper becomes a negative result showing that good resampling suffices, which is still worth reporting.