Looking Ahead in the Linear Stage of Hybrid Echo Cancellation

Hybrid echo cancellers split the work between two parts. A cheap classical adaptive filter learns the linear acoustic path from loudspeaker to microphone and subtracts the predicted echo. A small neural network then cleans up what is left: loudspeaker distortion, leaks, and noise. These systems are built causally. The adaptive filter only learns from the past, so every time the echo path changes (someone moves the phone, a door opens) or the near-end talker starts speaking over the far end, there is a stretch of time in which the linear filter is wrong. During that stretch the neural post-filter is handed a large residual it can only mask, and masking removes near-end speech along with the echo. Many real uses do allow some future context: meeting-recording cleanup, voice messages, and chunk-buffered calls that can tolerate tens of milliseconds of delay. When future audio is available, the obvious move is to make the neural post-filter bidirectional. We argue that the future is worth more upstream, inside the linear filter.

We propose to replace the causal adaptive filter with a Kalman smoother of the echo path. A Kalman smoother is the classical forward-then-backward estimator that revises each moment's estimate using later observations. Its noise statistics would be set frame by frame by a lightweight bidirectional linear recurrent network. The smoother carries information about the new path backward in time, so the transient after a path change mostly goes away. The bidirectional controller can see that near-end speech continues and freeze adaptation before the filter diverges. The core claim is testable: at matched compute and latency, future context placed in the linear stage should beat the same future context placed in the neural post-filter. The gain should be concentrated in the post-change and double-talk-onset windows, where causal hybrids fail.

Positioning Against Prior Work

The motivating evidence comes from hybrid filtering. Efficient High-Performance Bark-Scale Neural Network for Residual Echo and Noise Suppression cascades a subband NLMS canceller with a causal NSNet2 post-filter on 86 auditory (Bark-scale) bands. It reports 235 Million MACs/s against 2,170 for the end-to-end DeepVQE-S, and the best double-talk near-end preservation ("DT Other"). This shows the division of labor pays off, but both stages look only backward. HAD-ANC shows that an adaptive linear stage re-converges after an abrupt path inversion where pure deep models diverge. That result makes the reconvergence transient the natural target. We keep the linear stage for exactly that robustness and attack the transient itself.

The method draws on two lines of work that fuse past and future. A Bayesian Approach to Recurrence in Neural Networks derives recurrent gates as Bayesian posteriors and runs an exact forward filter and backward smoother. Its Bi-LBRU reaches 25.9% WER on AMI against 27.4% for a Bi-GRU. That work smooths abstract hidden states for speech recognition. We smooth a physical quantity, the echo-path taps, whose dynamics are genuinely close to linear. Neural Kalman Filtering for Speech Enhancement is the closest hybrid. It embeds an LSTM inside a Kalman predict-update loop and shows that the Kalman structure prevents over-suppression. Its state is the clean speech spectrum, and its recursion is strictly causal. Our state is the echo path, the recursion runs in both directions, and the network sets the filter's noise statistics rather than predicting the state itself.

For the controller we reuse the bidirectional linear recurrence of BLUR. BLUR uses diagonal, stable linear recurrences computed by parallel scans in linear time, with a nonlinear fusion of the two directions. LION plays a constraining role. It shows that naively summing forward and backward states double-counts the current frame, and it needs an explicit correction term. In the linear stage this issue is handled by construction, because the smoother's combination of forward and backward information is exact. We use LION's correction only in the ablation where the bidirectional model sits in the post-filter.

Core Mechanism

In each Bark band $b$ and frame $t$, the echo path is a short vector of filter taps $w_b(t)$ that drifts as a random walk. The microphone signal is the far-end reference $x_b(t)$ passed through that path, plus near-end speech and noise:

$$ w_b(t) = w_b(t-1) + q_b(t), \qquad y_b(t) = x_b(t)^\top w_b(t) + v_b(t). $$

Here $q_b(t)$ is the process noise, with variance $Q_b(t)$, and $v_b(t)$ is the near-end-plus-noise term, with variance $R_b(t)$. The two variances set how the filter behaves. A large $Q$ means "the path just changed, adapt fast." A large $R$ means "someone near-end is talking, don't trust this frame."

A bidirectional linear recurrent controller reads the reference and microphone band energies over the buffered chunk and outputs $Q_b(t)$ and $R_b(t)$. Its ingredients are BLUR-style diagonal recurrences in each direction, fused by a gated unit. A differentiable frequency-domain Kalman filter runs forward over the chunk. A Rauch–Tung–Striebel backward pass, the standard smoother for linear Gaussian models, then refines every $w_b(t)$ using later frames. The smoothed path gives the echo estimate to subtract. A Bark-band causal post-filter matching Seidel et al.'s NSNet2 handles nonlinear residuals.

The whole chain is trained end to end on a near-end reconstruction loss. We add a small auxiliary loss on the echo path wherever the simulated path is known. The lookahead is the chunk length $L$. It is swept from 0 frames, which gives a causal Kalman filter with a bidirectional-free controller, up to whole utterances. Compute stays linear in $L$ because both the controller scan and the Kalman recursion are linear-time.

Research Program

We build three things on one codebase. The first is the differentiable Kalman filter and smoother over Bark bands with a learned controller. The second is a matched family of variants that move future context between stages. The third is a transient-aware evaluation harness.

The harness is necessary because whole-clip averages dilute the effect we predict. We simulate echo paths from room impulse responses, following the Microsoft AEC Challenge 2023 augmentation recipe with nonlinear loudspeaker distortion used by Seidel et al. We insert path changes at known times: impulse-response swaps, gain jumps, and HAD-ANC-style polarity inversions. We also insert double-talk onsets at known times. Metrics are then reported separately in the first 0.5 s after each event and in steady state.

The analysis question is where future context helps and by how much. We expect the linear-stage gain to scale with how often and how abruptly the path changes. We expect the post-filter gain to be roughly flat.

Baselines

The naive reference is the linear canceller alone: causal subband NLMS with no neural stage. A second reference is the offline "two-pass" trick, which adapts NLMS over the whole clip and then re-filters with the final coefficients. That trick is the cheapest possible way to use the future, and it is ideal when the path never changes.

The strongest existing method is the Seidel et al. hybrid of subband NLMS with an 86-band NSNet2 post-filter. It is the best-documented system on the ICASSP 2023 AEC Challenge blind test set, and it is the design we are extending. Its reported figures are 235 Million MACs/s, a 0.22% real-time factor on a desktop CPU, and the top "DT Other" near-end preservation score. There is no reported numeric echo-quality score for it on Lacuna. We will therefore retrain it ourselves on the same 50,000 challenge-derived recordings, with the same Bark banding and the same tuning budget, and report its scores alongside ours.

We include DeepVQE-S as the end-to-end reference, stating the 2,170 Million MACs/s complexity reported in that comparison. No reported quality result was found on Lacuna for DeepVQE-S on this test set.

The key ablation is a two-by-two design. The linear stage is either a causal Kalman filter or our smoother. The post-filter is either causal or bidirectional, where the bidirectional version is a BLUR post-filter with LION's double-counting correction. Every cell gets the same lookahead $L$, the same total MAC budget within 10%, and the same number of hyperparameter trials. A further ablation keeps the smoother but fixes $Q$ and $R$ by grid search, which isolates the learned controller.

Beating the strongest baseline means two things. First, a statistically significant improvement over the retrained Seidel system in near-end preservation during double-talk on the blind set, at comparable MACs. Second, on the simulated set, a clear gain in echo reduction (ERLE) and near-end signal-to-distortion ratio (SI-SDR) in post-event windows over the "causal linear + bidirectional post-filter" cell at the same $L$.

Why not just…?

Why not just make the post-filter bidirectional and keep the causal adaptive filter? This is the real competitor. With enough context, a bidirectional mask can plausibly suppress transient residual echo well. Our argument is physical. A mask can only attenuate time-frequency cells, and during reconvergence the residual echo overlaps the near-end speech. Suppressing the echo therefore removes speech too. Subtracting with a correct path estimate does not. The two-by-two ablation settles it. If the "causal linear + bidirectional post-filter" cell matches our smoother in post-event windows at equal $L$ and compute, the core claim fails and we will report that.

Why not just run a classical Kalman smoother with hand-tuned noise statistics? Smoothing alone removes much of the path-change transient, so this ablation may capture most of the gain on clips with only path changes. It cannot resolve the conflict between fast tracking and stability during double-talk. One fixed $Q/R$ pair either chases near-end speech or tracks slowly. Neural Kalman Filtering for Speech Enhancement likewise gets its benefit from learned statistics inside a fixed structure. The fixed-statistics ablation measures exactly how much the learned bidirectional controller adds, separately for path-change and double-talk events.

Why not just adapt once over the whole clip and re-filter (two-pass NLMS)? For a static path, this is nearly optimal and costs almost nothing, so it is a baseline rather than a straw man. It assumes one path per clip. With a mid-clip change it averages the two paths and is wrong on both sides of the change. The smoother reduces to it when $Q$ is near zero, so the path-change set shows directly whether time-varying smoothing is needed.

Why not just train an end-to-end bidirectional network? Seidel et al. report a roughly tenfold compute gap between the hybrid and DeepVQE-S. HAD-ANC reports that pure deep controllers diverge under path inversion where the adaptive hybrid recovers. We will include the inversion events in our simulation so the same stress test applies to echo cancellation.

Decisive Checks

The experiment that settles the proposal is the post-event window analysis on the simulated set, crossed with the lookahead sweep. The claim holds if our system's advantage over the bidirectional-post-filter cell appears in the first 0.5 s after path changes and double-talk onsets, grows with $L$, and vanishes in steady state. That pattern would show the future is being used to fix the linear estimate rather than to add general capacity.

The blind-test result guards against simulation artifacts on real recordings, using near-end preservation in double-talk against the retrained Seidel system. The latency sweep answers the deployment question directly: it shows how many buffered milliseconds are needed before looking ahead in the linear stage pays for itself. ANC is out of scope because anti-noise must be emitted in real time, so no future context exists there.