Fast Envelopes, Slow Geometry: Kronecker Noise Covariances for Multi-Frame Beamforming in Bursty Noise

Microphone-array speech enhancement works by estimating how background noise is correlated across microphones, then building a linear filter (a beamformer) that cancels that correlated noise while passing the target voice. Filters that also look at a few past frames, rather than only the current one, are known to work much better in reverberant, changing scenes. The catch is size. With $M$ microphones and $K$ frames of context, the noise covariance is an $MK \times MK$ matrix. When noise changes quickly, as with door slams, passing buses, or wind, that matrix has to be estimated from only a handful of frames. The estimate then becomes singular, and the filter that has to invert it breaks down.

This note proposes a covariance model built around one observation: in bursty noise, different parts of the statistics change at different speeds. Where the noise comes from relative to the array, which sets the cross-microphone correlation, changes slowly. How loud the noise is from frame to frame changes very fast. The model writes the multi-frame noise covariance as a Kronecker product, meaning a large matrix built by multiplying a small microphone-by-microphone matrix with a small frame-by-frame matrix. The frame-by-frame factor is a stationary correlation pattern rescaled by a per-frame loudness envelope. Each piece is tracked at its own speed. The spatial factor and the stationary temporal correlation are averaged over long windows, where data are plentiful. The envelope has only $K$ numbers and can be re-estimated every frame. Because of the Kronecker structure, the inverse is cheap and always well-conditioned, which makes multi-frame beamforming with longer context feasible at frame rate.

Core Claim

Existing Kronecker spatio-temporal covariance estimators get their sample efficiency from forcing the temporal factor to be Toeplitz. A Toeplitz matrix is one whose entries depend only on the time lag between frames, which amounts to assuming the noise is stationary. Regularized Block Toeplitz Covariance Matrix Estimation via Kronecker Product Expansions builds sums of Kronecker products with Toeplitz temporal factors, plus a diagonal correction, Ledoit–Wolf shrinkage, and Tyler robustness. Its own conclusion warns that "if system dynamics change rapidly, this may introduce bias." Classification of Event-Related Potentials with Regularized Spatiotemporal LCMV Beamforming uses a single spatial-by-Toeplitz Kronecker product for EEG beamforming, fixed after calibration, and names covariance drift as its main limitation.

Our claim is that most of the non-stationarity that breaks these models is a time-varying gain, not a time-varying correlation shape. A modulated-Toeplitz temporal factor captures it while keeping nearly all of the parameter savings. The result should be that $MK$-dimensional multi-frame beamformers stay well-conditioned at context lengths where unstructured or shrunk estimates fail. Concretely, for frequency bin $f$ and frame $t$ the noise covariance of the stacked $K$-frame, $M$-microphone noise vector is modeled as

$$ \Sigma_f[t] \;=\; \big(D_t\, T_f\, D_t\big) \otimes S_f \;+\; I_K \otimes U_f , $$

The symbols are as follows. $S_f$ is an $M\times M$ Hermitian positive-definite spatial matrix normalized to unit trace. $T_f$ is a $K\times K$ Toeplitz correlation matrix with unit diagonal. $D_t=\mathrm{diag}(d_{t-K+1},\dots,d_t)$ holds the noise amplitude of each frame in the window. $I_K\otimes U_f$ is an optional diagonal sensor-noise floor, the diagonal correction from DC-KronPCA. These normalizations also resolve the scaling ambiguity of Kronecker factors, which Online change detection in SAR time-series with Kronecker product structured scaled Gaussian models handles with a unit-determinant constraint. Here, all absolute power is assigned to the envelope by construction, so each factor has a physical meaning and can be given its own forgetting rate. Without the floor term, the inverse is $(D_t^{-1} T_f^{-1} D_t^{-1}) \otimes S_f^{-1}$. The Toeplitz factor can be inverted with Levinson recursion, so the per-bin cost is roughly $O(M^3+K^2)$, compared with $O(M^3K^3)$ for an unstructured multi-frame matrix.

Relation to Prior Work

The closest structural relative is Mian et al. (Online change detection in SAR time-series with Kronecker product structured scaled Gaussian models). That work multiplies a Kronecker-structured covariance by one scalar "texture" per sample and tracks the factors with stochastic Riemannian gradient steps for radar change detection, assuming the background is piecewise stationary. Our envelope is a vector texture placed inside the temporal factor. Different frames within one filter's context window can therefore carry different power, which is exactly what happens when a burst starts mid-window. The target task, frame-rate beamforming, is also different from theirs.

The time-varying Kronecker graphical lasso of Interpretable Dynamic Network Modeling of Tensor Time Series via Kronecker Time-Varying Graphical Lasso tracks sparse precision factors under a smoothness penalty. Bursty acoustic noise violates that smoothness, and the method targets graph recovery rather than filtering. Robust Kronecker Product PCA for Spatio-Temporal Covariance Estimation absorbs outliers into a sparse matrix. A loud passing vehicle, however, is a structured, dense rescaling of whole frames, not a sparse corruption. Robust KronPCA also needs repeated singular value decompositions that are too slow for frame-rate use. We reuse its rearrangement idea only for an offline check of separation rank.

On the acoustic side, Sequential Multi-Frame Neural Beamforming for Speech Separation and Enhancement motivates the work. Multi-frame filtering gave an average 2.75 dB SI-SNR gain and a 14.2% absolute differential-WER reduction across its tasks, yet its covariances are plain mask-weighted averages with no structure. Statistical Beamformer Exploiting Non-stationarity and Sparsity with Spatially Constrained ICA for Robust Speech Recognition supplies a key ingredient in single-frame form. Its Laplacian model reweights each frame by an inverse power estimate, and online recursive least squares updates the weighted covariance. Our envelope $D_t$ is the multi-frame, Kronecker-compatible generalization of that weighting. Rank-1 Constrained Multichannel Wiener Filter for Speech Recognition in Noisy Environments shows that structural constraints on the speech covariance pay off on CHiME-4, but it stays narrowband and single-frame.

NICE-Beam: Neural Integrated Covariance Estimators for Time-Varying Beamformers appears to contradict the premise. There, unconstrained and Cholesky-parameterized LSTM covariance trackers beat more constrained estimators in dynamic scenes, which suggests that structure slows adaptation. We take this head on. The constraints that hurt there removed degrees of freedom the scene needed. Our model keeps the spatial factor full-rank and lets power move freely every frame. Whether that is "minimal enough" is precisely what the experiments test.

Research Program

The estimator runs per frequency bin on mask-weighted noise snapshots. The noise mask comes from one fixed, shared mask network, so mask quality does not differ across methods. Each frame performs one flip-flop pass, following the single-iteration result in the EEG beamforming paper.

The envelope is updated first, in closed form. Each frame's amplitude is the root of its mask-weighted noise power, after whitening by the current spatial factor and averaging over microphones and nearby bins. This step has no memory beyond a light floor.

Next, the envelope-normalized snapshots update the Toeplitz lag correlations with a medium forgetting factor. The spatial factor is then updated from snapshots whitened by the temporal factor, with a long forgetting factor. The rule is that the spatial factor keeps its long memory unless the envelope-normalized residual signals a direction change, in which case it switches to a short memory. This detector handles moving sources, the case that defeats pure time-scale separation.

The resulting covariance feeds a multi-frame MVDR filter or a multichannel Wiener filter (MCWF). The filter structure is identical across all methods. Only the noise covariance changes.

The program has three analysis pieces. The first is offline diagnostics on real noise recordings. We measure how much non-stationary energy the envelope explains: we compare the Kronecker-rank spectrum of the rearranged covariance before and after envelope normalization, using the KronPCA rearrangement from Covariance Estimation in High Dimensions Via Kronecker Product Expansions. If normalization does not concentrate the spectrum toward rank one, the core hypothesis is wrong and we will say so early. The second is a sensitivity study over context length $K$ (1 to 12 frames) and over separation rank 1 versus 2. The third is a controlled simulation with known ground-truth envelopes and spatial matrices. There we can measure covariance error and beamformer output signal-to-interference-plus-noise ratio against the number of available frames.

Baselines

The naive reference is a single-frame, narrowband MVDR beamformer whose noise covariance comes from mask-weighted recursive averaging with a tuned forgetting factor. This is the standard pipeline that every gain must exceed.

The first strong competitor is the unstructured multi-frame MCWF of Sequential Multi-Frame Neural Beamforming for Speech Separation and Enhancement, run in a single beamforming stage with our shared mask. We add the most competitive simple regularizer to it: Ledoit–Wolf shrinkage of the mask-weighted multi-frame covariance toward a scaled identity, with the shrinkage chosen analytically per bin. This is the strongest competitor on the conditioning question because it fixes singularity with no structural assumption.

The second strong competitor is the online weighted-covariance beamformer of Nakatani et al. (Statistical Beamformer Exploiting Non-stationarity and Sparsity with Spatially Constrained ICA for Robust Speech Recognition). It is the strongest existing method on CHiME-4 in the file, reporting 2.80% WER in batch mode against 3.18% for Mask-MVDR and 3.76% for basic MLDR. Because WER depends on the recognizer, we will not compare against 2.80% directly. We will rerun Mask-MVDR, the single-frame weighted beamformer, and all of our variants through one fixed recognizer back-end. The reported ordering, weighted single-frame beating Mask-MVDR, serves as a sanity check that our reproduction is faithful. Beating this baseline means a lower WER on the real CHiME-4 evaluation set under the same recognizer, with the gain holding in all four environments, not only on average.

The third competitor is NICE-Beam (NICE-Beam: Neural Integrated Covariance Estimators for Time-Varying Beamformers) on its spatialized DNS dynamic dataset. There it reports 6.80 dB SI-SDR for the "Arbitrary" model, against 4.04 dB for an oracle-informed buffer estimator, and PESQ 2.26 for the Cholesky model. We will retrain it on the same splits with a matched tuning budget. Beating it means exceeding its SI-SDR on the dynamic split while using no learned covariance parameters. A tie would still be notable, because our estimator has no trained parameters.

Three ablations isolate the key ingredient:

  • Kronecker with a plain Toeplitz temporal factor, with the envelope removed. This is the Greenewald–Hero and STBF-STRUCT model run online.
  • Kronecker with an unconstrained temporal factor.
  • An envelope-weighted multi-frame covariance with no Kronecker structure.

All methods share masks, frames, filter type, and hyperparameter search budget. Forgetting factors and shrinkage are tuned on the same development split.

Why not just…?

Why not just shrink the multi-frame covariance (Ledoit–Wolf or diagonal loading)? Shrinkage restores invertibility but does not share information between the spatial and temporal modes. In EEG, the Kronecker-Toeplitz estimator beat shrinkage under small calibration sets (Classification of Event-Related Potentials with Regularized Spatiotemporal LCMV Beamforming). On sensor-network anomaly detection, robust Kronecker estimation reached AUC 0.994 against 0.92 for the sample covariance (Regularized Block Toeplitz Covariance Matrix Estimation via Kronecker Product Expansions). Neither result involves bursty acoustic noise, so this alternative remains a real threat. If shrunk multi-frame covariances match our estimator at every $K$, the structural claim fails. The $K$-sweep is designed to show this either way.

Why not just use short-memory recursive averaging? A short forgetting window tracks bursts but forces the spatial estimate to be rebuilt from a few frames as well. That is the sample-starvation failure the file describes, and NICE-Beam's buffer baseline reached only 4.04 dB in dynamic scenes even with oracle separation. Our model's whole point is to decouple these two memories. The single-frame recursive baseline and the envelope-free Kronecker ablation together show whether decoupling, rather than structure alone, carries the gain.

Why not just reweight frames by power, as the Laplacian beamformer already does, without any Kronecker structure? This is the closest simple alternative, and the envelope-weighted unstructured ablation tests it exactly. Our prediction is that the two match at $K\le 2$, where $MK$ is small enough to estimate directly. Kronecker structure should matter only once $MK$ exceeds the effective number of frames. If no such crossover appears, the contribution reduces to "multi-frame Laplacian weighting," and we will report it as that.

Why not just learn the covariance with a neural tracker, or skip beamforming altogether? NICE-Beam is included as a baseline, and the end-to-end question is out of scope. Linear filtering is kept because the CHiME-4 results in Rank-1 Constrained Multichannel Wiener Filter for Speech Recognition in Noisy Environments and Nakatani et al. indicate that distortion-free linear filters help recognizers. A useful hybrid, if ours wins, would have NICE-Beam output only the envelope.

Proposed Experiments

The evaluation uses three real or realistic benchmarks plus a controlled one:

  • CHiME-4 (6-channel tablet; bus, cafe, pedestrian, street): WER through one fixed recognizer.
  • Spatialized DNS static and dynamic sets from NICE-Beam: SI-SDR and PESQ.
  • LibriCSS (7-channel, 0–40% overlap): concatenated minimum-permutation WER, using the single-stage multi-frame setup. No numeric LibriCSS result is reported for the multi-frame baseline in the material, so all numbers there will be our own reruns.
  • TIMIT mixed with jackhammer, bus, combat, and babble noise, spatialized with simulated room responses, following the noise types in Regularized non-negative matrix factorization with temporal dependencies for speech denoising. Here we control envelope burstiness directly.

Four checks decide the claim. First, as $K$ grows, the unstructured and shrunk multi-frame beamformers should plateau or degrade while ours keeps improving. The gap should widen with $MK$ relative to the effective sample count. Second, when test segments are stratified by envelope variance, our advantage over the plain-Toeplitz Kronecker ablation should concentrate in high-variance (bursty) segments and vanish in stationary ones. If it does not, the envelope is not doing the work. Third, on the dynamic DNS set, which has head rotations, the change-triggered spatial memory should recover performance that pure long memory loses. Comparing the two isolates the moving-source failure mode. Fourth, we predict the method helps least on babble. Babble shares spatial-temporal statistics with speech, so masks leak and no covariance model can repair corrupted snapshots. We will report this case separately rather than average it away. Wall-clock cost per frame against the unstructured multi-frame filter will also be reported, since real-time feasibility is part of the claim.