Squeezing Clock Drift Through a Narrow Channel: Drift-Robust Echo Cancellation Under Double-Talk

When a smart TV plays sound through a wireless soundbar and listens with its own microphones, the two devices run on separate crystal clocks. Their clocks disagree by tens to hundreds of parts per million. Echo cancellers subtract a predicted copy of the loudspeaker signal from the microphone, so they need the reference signal and the recording to stay aligned to within a sample. That alignment slowly slides away. The problem is worst during double-talk, when the local person speaks over the remote one. The canceller then cannot re-measure the alignment, because the local speech swamps the echo, yet the clocks keep drifting. When the local talker stops, the canceller is out of phase and echo bursts through. Neural echo suppressors facing the same confusion tend to mute the local talker.

We propose to build an echo suppressor in which all timing information about the reference must pass through a tiny, deliberately constrained channel. That channel is a few numbers per frame, offset and drift rate. It is penalized for carrying anything the physics of clock drift does not already predict. This is an information bottleneck: a training penalty that limits how much information a learned internal code carries about its input, so that only what matters for the task survives. Here the bottleneck is aimed at one nuisance with known structure, the time warp between two clocks, rather than applied to the whole network. The expected payoff is a suppressor that keeps tracking drift through long double-talk bursts, extrapolates to drift rates it never saw in training, and exposes an interpretable offset estimate. It should also fit in the compute budget of a lightweight post-filter.

Core Claim

The obvious combination of these three topics is to add a generic variational bottleneck loss to a recurrent echo suppressor trained on drifting audio. We argue this is the wrong design. The suppressor needs alignment information to subtract echo. Compressing the whole latent code therefore trades echo removal against robustness, which is the "compression paradox" the file identifies. Our claim is that the paradox disappears if compression is targeted. Alignment is routed through a dedicated low-dimensional state $z_t$ that warps the reference signal explicitly. That state is charged an information cost measured against a drift-dynamics prior. The suppression network then sees an already-aligned reference and gets only a short lag window of its own. Under this design, double-talk becomes a regime where the timing state is expected to "coast": the posterior widens, the state follows the prior's constant-drift prediction, and it re-locks when single-talk returns. We expect this to hold without a hand-built double-talk detector and without hand-derived skew approximations.

The training objective is

$$\mathcal{L} = \mathbb{E}\,\big\|S - \hat S\big\|^2 + \beta \sum_t \mathrm{KL}\!\left(q(z_t \mid d_{\le t}, x_{\le t}, z_{t-1}) \,\big\|\, p(z_t \mid z_{t-1})\right).$$

The symbols are as follows. $S$ and $\hat S$ are the clean and estimated near-end spectra. $d$ is the microphone signal and $x$ is the far-end reference. $z_t = (\tau_t, \delta_t)$ holds the offset and drift rate at frame $t$. $q$ is the learned timing encoder. The prior $p(z_t \mid z_{t-1}) = \mathcal{N}\big((\tau_{t-1}+\delta_{t-1},\ \delta_{t-1}),\ \mathrm{diag}(\sigma_\tau^2, \sigma_\delta^2)\big)$ encodes the linear-drift model $\Delta t(t) = \tau_0 + \delta\, t$. The KL term upper-bounds the information the timing channel takes from the observations beyond what the drift dynamics already predict. The weight $\beta$ sets how much the tracker trusts noisy evidence versus the prior.

Relation to Prior Work

Three existing systems each solve part of this problem and each leaves the double-talk-under-drift case open.

Clock Skew Robust Acoustic Echo Cancellation tracks skew with subband Kalman filters over a multi-hop transform, using a Huber loss to resist double-talk. It shows that conventional adaptive filters remove under 1 dB of echo at 2 Hz skew (about 125 ppm). Its limits are a linearized block-skew approximation, high cost from small hop sizes, and linear-only echo modeling. We keep its state-space intuition but learn the tracker end to end, jointly with a nonlinear suppressor.

Elucidating Clock-drift Using Real-world Audios In Wireless Mode For Time-offset Insensitive End-to-End Asynchronous Acoustic Echo Cancellation (SY4) trains a convolutional recurrent network with an LSTM middle layer to absorb wireless drift implicitly under MSE loss. It reports about 2 dB average ERLE improvement at SERs from −15 to −35 dB. Its drift handling is a black box, with no mechanism that forces it to persist through double-talk or generalize beyond the training drift range. That is exactly what we test.

Cross-Talk Reduction absorbs small jitter with non-causal look-ahead taps and concedes that significant drift degrades them. This constrains our design. A bounded lag window can absorb only residual misalignment, which is why the suppressor gets only a short one and bulk drift must go through $z_t$.

The method-supplying precedent is Unsupervised Speech Decomposition via Triple Information Bottleneck. SPEECHFLOW shows that timing information can be isolated by routing it through a designated narrow channel while other paths are denied it through random time resampling. Clock drift is effectively an involuntary version of that resampling. We borrow the idea of routing timing through a designated channel, but put a physical dynamics prior on that channel instead of treating factors as independent, which is the independence assumption SPEECHFLOW's authors identify as a weakness.

The IB principle itself comes from Tishby et al.. Yao et al. motivate the concern that unconstrained networks latch onto nuisance cues. They showed that a variational bottleneck made an ultrasound classifier ignore calibration markers. Their bottleneck was generic because the nuisance had no structure, whereas ours is tailored because drift does.

On the suppression side, the backbone budget follows Efficient High-Performance Bark-Scale Neural Network for Residual Echo and Noise Suppression, which reports 235 million MACs/s. We also follow Efficient Acoustic Echo Suppression with Condition-Aware Training (gGCRN16), which found 100% double-talk training best for preserving near-end speech. That finding is in tension with classical skew trackers, which need single-talk periods. Our coasting prior is one proposed resolution, and we test it directly. Both of these systems, and Joint Neural AEC and Beamforming with Double-Talk Detection and Nonlinear Residual Echo Suppression Based on Multi-Stream Conv-TasNet, assume a synchronized reference.

Research Program

The system has three parts.

The first is a causal timing encoder. It reads cheap cross-correlation features between the microphone and the reference over a coarse lag range, plus its previous state. From these it outputs a Gaussian posterior over $(\tau_t, \delta_t)$.

The second is a differentiable fractional-delay resampler. It warps the reference by the sampled $\tau_t$ at the sample level, before any Bark-band pooling. This matters because pooling into coarse bands destroys the fine phase detail that fractional drift lives in.

The third is a lightweight suppression network in the style of gGCRN16 or the Bark-scale post-filter. It receives the microphone signal and the warped reference with a lag window of only a few frames, so it cannot re-derive large misalignments itself.

Training data come from AEC Challenge far-end and near-end material, including the INTERSPEECH 2021 Acoustic Echo Cancellation Challenge corpus. We inject synthetic clock warps by resampling the reference with controlled offsets, constant skews, and slowly varying skews, so ground-truth $z_t$ is known. Double-talk proportion is a controlled variable.

The analysis side sweeps $\beta$ to trace the trade-off between agility, meaning re-locking after sudden offset jumps such as Bluetooth buffer resets, and robustness, meaning coasting through double-talk. It also inspects whether the posterior variance of $z_t$ rises during double-talk without being told when double-talk occurs.

Baselines

The naive reference points are the unprocessed microphone signal and a synchronous-assumption linear canceller (a partitioned-block frequency-domain Kalman filter) run on drifting data. For the latter, Clock Skew Robust Acoustic Echo Cancellation reports under 1 dB ERLE at 2 Hz skew on its real-recording dataset. Any method must clear that by a wide margin.

The strongest classical competitor is that paper's SRK Kalman tracker. It is the only explicit, published drift tracker for echo cancellation that also resists double-talk. We run it with its reported best hop size of 40 and a neural post-filter on its output so it is not handicapped on nonlinear echo. We report its compute alongside ours. The file gives no absolute ERLE for SRK beyond "significant echo suppression", so we will reproduce it rather than quote a number.

The strongest neural competitor is SY4 from Elucidating Clock-drift…. Its reported gain is about 2 dB average ERLE over its own baselines on its wireless SET-1 to SET-3 recordings. No absolute ERLE value was found on Lacuna. We retrain it on identical data with identical drift augmentation, matched parameter count and MACs, and the same hyperparameter search budget.

The cascade baseline has the same compute envelope as ours. It pairs a non-learned drift estimator and resampler with the same suppression backbone, followed by the Bark-scale post-filter.

For the synchronized upper bound, we use the Bark-scale post-filter and gGCRN16 fed the ground-truth-aligned reference. For calibration on the double-talk quality scale, TasNet-MI reports double-talk PESQ of 2.50 at −18.2 dB SER on TIMIT+MUSAN without drift.

Three ablations isolate the key ingredient. The first replaces the drift prior with a standard-normal prior, which is a generic variational bottleneck. The second removes the KL term entirely, leaving an unconstrained timing channel. The third widens the suppressor's lag window so alignment can bypass $z_t$.

Beating the baselines means the following. Our model must significantly outperform drift-augmented SY4 and the cascade on echo leakage after long double-talk bursts, and on out-of-range drift rates. It must stay within 1 dB ERLE and 0.1 AECMOS of the synchronized upper bound in-distribution. It must also cost well below SRK in compute.

Why not just…?

Why not just train an ordinary recurrent suppressor with heavy drift augmentation? This is the most serious threat. SY4 already shows implicit absorption works in-distribution, and augmentation is free. Nothing in the file shows how an implicit recurrent state behaves when drift continues through several seconds of double-talk, or at drift rates outside training. If the drift-augmented SY4 matches us on the post-double-talk leakage test and the extrapolation test, the bottleneck is unjustified and we will say so. The ablation without a KL term shows how much of any gain comes from the explicit warp alone versus the rate penalty.

Why not just estimate the sampling-rate mismatch and resample, then run a standard synchronous canceller? This cascade is essentially our architecture without end-to-end training and without the learned coasting behavior, which is why we include it at equal compute. The file shows that frozen or naively updated trackers lose phase during double-talk: when a detector freezes adaptation, the clocks keep drifting. We expect the cascade to be competitive in single-talk and to fail specifically on echo bursts after double-talk. If it does not fail there, our contribution shrinks to interpretability.

Why not just use the SRK Kalman filter? It is principled, but its skew model is a linearization, its fine hop size makes it expensive, and it is linear in the echo path. We compare against it directly at reported settings, with a post-filter added. We count a win only if we match its drift tracking at a fraction of its compute while handling loudspeaker nonlinearity.

Why not just add a generic information bottleneck, or dropout and weight decay? These penalize all information equally, including the alignment the canceller needs. The standard-normal-prior ablation tests this directly. We predict it shows a monotone loss of ERLE as $\beta$ grows, with no gain in double-talk robustness.

Proposed Experiments

We evaluate on four sources. The first is the Helwani et al. Interspeech 2022 skew dataset used by the SRK paper. The second is the wireless asynchronous SET-1 to SET-3 recordings from SY4, if they can be obtained; otherwise we reproduce its protocol with our own soundbar recordings. The third is the ICASSP 2023 AEC Challenge blind test set with injected warps. The fourth is our synthetic set with known ground-truth drift.

Metrics are ERLE in far-end single-talk and PESQ in double-talk. We add AECMOS, because the INTERSPEECH 2021 challenge found ERLE and PESQ correlate poorly with listener ratings for delayed echo. We also measure echo leakage in the two seconds after each double-talk burst ends, offset-tracking error against ground truth, and MACs per second.

Three checks decide the outcome.

The first is a coasting test. Double-talk bursts of 2 to 10 seconds are inserted under constant 60 to 125 ppm drift. The proposal succeeds if its post-burst leakage stays flat as burst length grows while SY4 and the cascade degrade.

The second is an extrapolation test. Models are trained at up to 60 ppm and tested at 125 and 200 ppm.

The third is a condition-split test in the style of gGCRN16. Models are trained with 100% double-talk data to check whether the dynamics prior removes the need for single-talk periods that classical trackers depend on.

Across all three, the $\beta$ sweep should show an interior optimum only for the physics prior. If the generic prior also shows that optimum, the targeted-bottleneck claim is false.