A speech recognizer trained on thousands of hours still makes many more errors on a new speaker in a noisy room. It can adapt by adjusting a few small parameters from that speaker's first utterances, but a handful of seconds of audio is easy to overfit. Bayesian adaptation fixes much of this. Instead of committing to one best guess for the adapted parameters, it keeps a probability distribution over them, which stays wide when evidence is thin. The existing methods have two weaknesses. They approximate that distribution as fully independent per parameter (a "mean-field" approximation), and they re-run an iterative optimization over all accumulated audio rather than updating once as each utterance arrives. This note proposes splitting the adaptation parameters into small, explicitly separate modules: one per layer, with speaker and acoustic environment kept apart. Each module then keeps a full-covariance belief, meaning a distribution that also tracks how its parameters vary together. That belief is updated in a single Kalman-filter step per utterance, and each module has its own rate of drift.
The bet is that the module boundaries do two jobs at once. They make correlated beliefs cheap enough to update online. They also give each source of variation its own assumption about how fast it changes: a speaker's voice is stable, while background noise can switch without warning. If the bet holds, a streaming recognizer should adapt faster in its first few utterances than mean-field Bayesian methods, and should recover from a sudden noise change without forgetting what it learned about the speaker.
Core Claim
Our hypothesis is that in the few-utterance regime, the correlations between parameters inside a small adaptation module carry useful information that mean-field posteriors throw away. Correlations across modules matter much less. If so, a block-diagonal posterior aligned with the module structure gets most of the benefit of a full posterior at a small fraction of the cost. A second hypothesis is that giving speaker and environment modules separate state dynamics lets the filter handle abrupt environment shifts. This is the failure that Single-Step Online Adaptation of Modular Bayesian Deep Receivers with Streaming Data reports for its single smooth Gaussian random-walk model.
Concretely, let the adaptation parameters be partitioned into modules $k = 1, \dots, K$ with $P_k$ parameters each. Module $k$ carries a Gaussian belief with mean $\mu_k$ and full covariance $\Sigma_k$. Before each utterance, the belief is widened by a module-specific drift covariance $Q_k$. The belief is then corrected using the gradient and a Gauss-Newton curvature estimate of the recognition loss on that utterance's first-pass hypothesis. Storage and update cost scale as
$$ \sum_{k=1}^{K} P_k^2 \quad \text{rather than} \quad \Big(\sum_{k=1}^{K} P_k\Big)^2 , $$
because a full covariance over $P$ parameters needs $P^2$ entries. The drift $Q_k$ is small for speaker modules. For environment modules it is larger, and it is inflated, or the module is reset to its prior, when a simple likelihood-drop detector fires.
How This Relates to Prior Work
Two papers supply the method.
Bayesian Learning for Deep Neural Network Adaptation learns variational distributions over hidden-unit scaling vectors (BLHUC), bias vectors (BHUB) and activation slopes (BPAct). It beats deterministic adaptation from as few as five utterances. Its posteriors are mean-field and fitted in batch. We keep its adaptation transforms but replace both the posterior family and the update rule.
Single-Step Online Adaptation of Modular Bayesian Deep Receivers with Streaming Data contributes the single-step filters (CM-EKF, VD-EKF, Lo-Fi) for tracking network weights. It also contributes the key structural observation: full-covariance tracking runs out of memory on monolithic networks but is fast on small modular sub-networks of fewer than 500 parameters. We carry that insight from wireless receivers to speech recognition. Our modules are adaptation transforms attached to a frozen recognizer, not an unfolded signal-processing pipeline. We also address the abrupt-shift failure that paper leaves open.
Factorised Speaker-environment Adaptive Training of Conformer Speech Recognition Systems motivates the factorisation. It splits adaptation into speaker and environment transforms, LHUC and HUB, and applies variational Bayes to them on Switchboard mixed with WHAM! noise. Its updates are offline, its posteriors are mean-field, and both factors share one notion of time. Our proposal differs in all three respects.
Two papers constrain the design space. Modular Domain Adaptation for Conformer-Based Streaming ASR attaches deterministic per-domain modules to a frozen streaming Conformer and finds that feed-forward blocks carry domain knowledge and that parallel adapters beat sequential ones. Its data are hundreds of thousands of hours per domain, which tells us those large modules are the wrong unit for seconds of audio. Your representations are in the network shows that frozen-backbone adapters can be trained without backpropagating through the backbone. That makes cached-activation updates attractive, although its training is deterministic.
The modular software line plays a supporting role. DeepDIVA and UCEE argue that fixed interfaces between components are what make comparisons fair. We adopt the same discipline so that posterior family, module partition and update rule can each be swapped while everything else stays fixed. We claim no novelty for the software itself.
Research Program
We will build a small adaptation library around one Conformer recognizer. The recognizer is trained on the 300-hour Switchboard-1 setup with WHAM! noise, following Deng et al. The library has four interchangeable parts:
- the backbone hook, which decides where transforms attach;
- the module partition: per-layer LHUC, per-layer HUB, speaker versus environment, or one global block;
- the belief family: point estimate, diagonal, block-full, or full;
- the update rule: batch variational EM, multi-step SGD, or single-step EKF.
The backbone is frozen and its activations at the attachment points are cached per utterance. Each update therefore touches only the adaptation parameters.
The analytical work asks when within-module correlations should matter. We will measure the posterior correlation structure of the full-covariance filter after a few utterances, comparing correlation mass within modules against mass across modules. This lets us check whether the module boundaries we chose by hand match where the correlations actually are. Because LHUC parameter counts are modest, full covariance over all adaptation parameters is affordable here. That turns the full-covariance filter into a genuine upper-bound comparison rather than a hypothetical one, and lets us test the block approximation directly. We will then repeat the comparison with larger low-rank parallel adapters, where full covariance becomes impractical, to show where the module structure becomes necessary rather than merely convenient.
Baselines
The naive reference is the unadapted Conformer scored on every utterance. Every adaptive method must beat it on the same streams.
The strongest existing method is variational Bayesian cascaded factorised adaptation from Factorised Speaker-environment Adaptive Training of Conformer Speech Recognition Systems. That paper reports up to 3.1% absolute (10.4% relative) WER reduction over the unadapted Conformer, and up to 0.8% absolute over deterministic factorised adaptation, across Hub5'00, RT02 and RT03. For the few-utterance regime, the reference figure is BLHUC from Bayesian Learning for Deep Neural Network Adaptation: up to 1.4% absolute (7.2% relative) WER reduction on Hub5'00 CallHome using the first five utterances. Those numbers come from a CNN-TDNN, so we will re-run BLHUC on our Conformer rather than compare across systems. We will run Bayesian CFA in a streaming protocol, re-fitting it on all audio seen so far after each utterance, and give it the same first-pass hypotheses and the same priors. Its iteration count will be tuned on a held-out development speaker set with the same budget we give our own hyperparameters ($Q_k$ and the detector threshold). Beating it means lower cumulative WER over the first one to ten utterances per speaker at equal or lower update compute, and no worse WER at convergence.
Deterministic LHUC and deterministic CFA, trained by a few SGD steps per utterance with KL regularization toward the unadapted model, represent the cheap non-Bayesian route. They get the same development tuning budget.
Two ablations isolate the key ingredient:
- Covariance structure. A VD-EKF that uses the identical single-step filter with a diagonal covariance, and a full-covariance EKF over all parameters. The difference between diagonal and block-full measures the value of within-module correlations. The difference between block-full and full measures what the module boundaries cost.
- Separate dynamics. A variant with one shared drift term for all modules, which tests whether separating speaker and environment timescales is what handles abrupt shifts.
Why not just…?
Why not just use deterministic LHUC with KL regularization and early stopping? Wang et al. report that BLHUC consistently outperforms deterministic LHUC, MAP adaptation, f-MLLR and KL-divergence regularization on Hub5'00 when adapting from the first five utterances. Deng et al. likewise report up to 0.8% absolute gain of Bayesian over deterministic factorised adaptation. Both results come from batch settings, so we include the tuned deterministic baseline in the streaming protocol rather than assume the gap carries over.
Why not just re-run batch variational BLHUC or Bayesian CFA after each utterance? It may match our accuracy. If it does, the remaining argument is cost. Re-fitting grows with the amount of audio accumulated, whereas a filter step is constant per utterance; the receiver paper reports 0.103 ms per sample for VD-EKF on a CPU, though in a very different setting. Batch re-fitting also has no natural way to forget a stale environment. The environment-switch streams below will show whether that matters in practice.
Why not just use a diagonal Kalman filter over all adaptation parameters, with no modules at all? This is the real threat. It keeps single-step streaming and linear cost and drops the module claim entirely. If diagonal VD-EKF matches block-full filtering in early-utterance WER and recovery time, we will conclude that within-module correlations do not matter at this scale. The contribution would then shrink to streaming with factor-specific dynamics, which the shared-drift ablation tests separately. We will report either outcome.
Proposed Experiments
Streams are built from Hub5'00 (SWBD and CHE), RT02 and RT03 conversation sides. Each speaker's utterances are presented in order with WHAM! noise mixed in. Partway through each stream the noise type and level switch abruptly, in both seen and unseen noise conditions. Adaptation is unsupervised, using first-pass hypotheses. For extreme scarcity we add UASpeech, where Recent Progress in the CUHK Dysarthric Speech Recognition System reports Bayesian LHUC adaptation from 3.06 seconds of speech within a system reaching 25.21% WER. That system includes architecture search and data augmentation, so we compare against our own re-run BLHUC rather than that headline number.
We report four metrics:
- WER as a function of the number of utterances seen;
- cumulative stream WER;
- recovery time after a noise switch, measured as the number of utterances needed to return within a set margin of the pre-switch WER;
- per-update latency and memory.
Three outcomes are decisive. First, block-full filtering must beat the diagonal filter over the first five utterances on CHE while staying within noise of the full-covariance filter. Second, separate per-factor dynamics must cut post-switch recovery time relative to shared dynamics without raising speaker-module error before the switch. Third, both properties must hold on the larger parallel-adapter configuration, where full covariance can no longer serve as an upper-bound comparison. If the first check fails, we will drop the modular-covariance claim and report the result as a streaming factorisation study.
References
- Single-Step Online Adaptation of Modular Bayesian Deep Receivers with Streaming Data
- Bayesian Learning for Deep Neural Network Adaptation
- Factorised Speaker-environment Adaptive Training of Conformer Speech Recognition Systems
- Modular Domain Adaptation for Conformer-Based Streaming ASR
- Your representations are in the network: composable and parallel adaptation for large scale models
- DeepDIVA: A Highly-Functional Python Framework for Reproducible Experiments
- Universal Causal Evaluation Engine: An API for empirically evaluating causal inference models
- Recent Progress in the CUHK Dysarthric Speech Recognition System