Model-based speech enhancement often describes a noisy recording as several sound sources that add together, such as a talker, a street, and a ventilation fan. Each source gets its own hidden Markov model (HMM), a model that moves between a small set of hidden "states", each with its own typical spectrum. The standard way to combine them builds one joint model whose states are every pairing of speech state and noise state. That works for two sources but grows multiplicatively with each source added. It also means nothing can be reused: adding a fan means rebuilding and re-deriving the whole system. Engineering disciplines handle the same problem with hierarchical composition. Blocks hide their insides behind a fixed interface, and a group of connected blocks can itself be wrapped up and used as one block.
This note proposes that interface for HMM sound sources. Each source component sees only the share of the observed spectrum currently attributed to it. It runs its own forward-backward pass, the standard two-pass HMM inference, on that share, and reports back a predicted spectrum. A composite, for example a "street scene" made of traffic plus babble with a slow scene-level chain deciding which is active, exposes exactly the same interface. Composites can therefore nest to any depth. Inference cost grows with the sum of the component state counts, not their product. A component trained once can be added to a new mixture without retraining the others. The open empirical question is whether the accuracy lost by not tracking every joint state is small enough to be worth these gains. The program is built to answer that.
Core Claim
The interface rests on one fact about Poisson emissions, which An NMF-HMM Speech Enhancement Method Based on Kullback-Leibler Divergence already uses for a single speech HMM. In that work, each state emits magnitude-spectrum "counts" as a sum of Poisson variables, so that learning reduces to the familiar multiplicative non-negative matrix factorization (NMF) updates. A sum of independent Poisson sources is itself Poisson. Given each source's predicted rate, the expected share of an observed count that belongs to source $c$ therefore has a closed form:
$$\hat v^{c}_{ft} = v_{ft}\,\frac{\tilde\lambda^{c}_{ft}}{\sum_{c'} \tilde\lambda^{c'}_{ft}}.$$
Here $v_{ft}$ is the observed magnitude in frequency bin $f$ and frame $t$. The quantity $\tilde\lambda^{c}_{ft}$ is source $c$'s predicted rate, taken as the exponentiated posterior mean of its log rate under that component's current state posterior.
Each component's whole contract is the following exchange. It receives $\hat v^{c}$, runs forward-backward over its own $J_c$ states, and returns $\tilde\lambda^{c}$ together with its log evidence. Iterating this exchange is a coordinate-ascent variational scheme. Its per-iteration cost is $O(T\sum_c J_c^2)$, where $T$ is the number of frames. Exact joint-state inference costs $O(T\prod_c J_c^2)$, the Cartesian-product growth that Model-based noise suppression using unsupervised estimation of hidden Markov model for non-stationary noise accepts for one speech HMM and one noise HMM.
Two features go beyond a flat factorial HMM, which is a known model.
The first is closure under composition. A composite component has a slow gating chain over which children are active, with time-scale separation in the spirit of Infinite Hierarchical Hidden Markov Models. It splits its incoming share among its children using the same formula, weighted by its gating posterior, and returns the gated sum of their rates. To its parent it looks like an atomic component.
The second is post-hoc reuse. Components are trained in isolation, and at test time some may be left free to adapt with no supervision, as in Fujimoto and Nakatani's noise HMM. A library of pretrained components can then be assembled into mixtures never seen during training.
The claim is falsifiable. On two-source problems, the component interface should match exact joint inference within a small margin. On three or more simultaneous sources, where exact joint inference is impractical, it should beat a single flat noise HMM trained on the pooled noise.
Relation to Prior Work
Three papers supply the method.
The NMF-HMM paper supplies the Poisson emission model and its update rules. It composes only one speech HMM with fixed noise bases, and the file notes it degrades under mismatched noise.
Model-based noise suppression using unsupervised estimation of hidden Markov model for non-stationary noise supplies the idea of an unsupervised ergodic noise HMM, one in which any state can follow any other, fitted by Baum-Welch. It is also the clearest evidence that noise dynamics matter, since it lowers average WER from 33.0% with a static Gaussian mixture to 29.8%. Its monolithic product construction is exactly what this proposal replaces.
Coupling identification and reconstruction of missing features for noise-robust automatic speech recognition supplies further motivation. Its word-level HMM imputation reaches 75.42% accuracy at −6 dB on CHiME, against 70.50% for static imputation, which shows that temporal priors help even at very low SNR.
Two papers come closest structurally.
Online Single-Microphone Source Separation using Non-Linear Autoregressive Models already builds plug-and-play source blocks in a factor graph. A factor graph is a diagram of variables and the local functions linking them, over which inference runs by passing messages. Its sources are continuous neural autoregressive processes, so it has no discrete regimes. The authors also state that backward (smoothing) messages require invertible blocks, which they do not have. Discrete HMM components avoid that obstacle, because forward-backward over a fixed lag is exact and cheap.
GraphPPL.jl: A Probabilistic Programming Language for Graphical Models provides syntactic model nesting and reactive message passing. Its dynamic examples are continuous Gaussian filters on synthetic data. The proposal implements its components as GraphPPL submodels so that nesting comes for free, but it adds what that paper does not provide: discrete switching components, a specific closed interface, and audio benchmarks.
Two other papers constrain or frame the work. A theoretical analysis of speech recognition based on feature trajectory models shows that HMMs with dynamic-feature constraints amount to fixed-lag Kalman smoothing. This justifies giving every component a fixed-lag smoothing mode rather than filtering only. Slice-and-Pack: Tailoring Deep Models for Customized Requirements composes frozen neural fragments but needs repair layers when shared representations break. It serves as a warning: an interface that is too narrow causes "ability collapse". The proposal must therefore measure interface loss directly rather than assume it away.
Research Program
The first piece is the component itself. It is a Poisson-emission HMM with a basis matrix per state, trained by the NMF-HMM updates. It has three inference modes: full-utterance forward-backward, fixed-lag smoothing with lag $L$, and forward-only filtering. A library of components will be trained once: a speech HMM on TIMIT using the 40-state setting Xiang et al. found best, and one HMM per NOISEX-92 noise type (babble, F16, factory, white) plus MS-SNSD air conditioner. Each component also has an "open" variant whose bases and transitions adapt online by Baum-Welch, following Fujimoto and Nakatani.
The second piece is the composite. It consists of a set of children plus an optional gating chain whose states are subsets of active children. The gating chain has slow transitions and is learned or set by hand. The composite implements the same receive-share, return-rate contract as an atomic component.
The third piece is the analysis. First, how far the iterated split falls short of exact joint inference on small problems. Second, how that shortfall changes with the number of sources and their spectral overlap. Third, whether nesting with gating beats a flat set of the same children. Without gating, nesting is mathematically identical to flat composition, so any difference isolates the gating chain's contribution.
Enhanced speech is reconstructed as a Wiener-style mask built from the speech component's share.
Baselines
The naive reference is the unprocessed mixture. On LibriSpeech plus MS-SNSD this is input SNR near −0.16 dB, as reported by van Erp and de Vries. On TIMIT plus NOISEX-92 it is the 0, 5 and 10 dB mixtures used by Xiang et al.
The static reference is frame-independent NMF with the same speech and noise bases but no transitions. Xiang et al. report that adding the HMM gains over 7% STOI relative to NMF-based methods. The file gives no absolute value, so that number will be re-measured here.
The first strongest competitor is exact joint-state composition in the style of Fujimoto and Nakatani. It is the strongest because it is what the interface approximates; if the interface loses badly to it, the proposal fails. It will use identical components and identical initialization. Pruning will be added once the joint state count exceeds what fits in memory. The reported reference is 29.8% average WER on IPA-98 in Model-based noise suppression using unsupervised estimation of hidden Markov model for non-stationary noise. That corpus may be unavailable, so the comparison will be re-run on the shared benchmarks below. Beating it means coming within 0.1 PESQ and 1 point STOI on two-source mixtures while running at least an order of magnitude faster with four sources. With three or more sources, beating it means outperforming its pruned version.
The second strongest competitor is the modular continuous-source factor graph of Online Single-Microphone Source Separation using Non-Linear Autoregressive Models. It reports a 4.59 dB SNR gain on 100 LibriSpeech utterances with MS-SNSD air-conditioner noise. That setup will be reproduced exactly, and the proposal must exceed 4.59 dB in forward-only mode to count as better at matched latency.
Two further baselines complete the set. One is the flat pooled model: a single large noise HMM trained on all noise types together, with the same total number of states as the library. The other is a deep reference: the U-Net configuration from Incorporating Symbolic Sequential Modeling for Speech Enhancement, which reports 2.126 PESQ and 0.725 STOI on TIMIT with 100 noise types, retrained here on the same training noise as the library. All baselines get the same training audio and the same budget of hyperparameter trials, 20 each, selected on a held-out development split.
Why not just…?
Why not just keep the exact joint model and prune it? For two sources with small state counts this may be as good or better, and the proposal concedes that case if the experiment shows it. Beam pruning becomes fragile when several sources overlap spectrally, because the right joint state can easily fall off the beam. The source-count sweep will show the crossover point directly.
Why not just train one bigger noise HMM on every noise you expect? This is the most serious threat, because it is simple and keeps exact two-way inference with speech. It cannot represent two noises happening at once, except by spending states on every combination, and it must be retrained whenever a new noise type is added. The flat pooled baseline is designed to test this. If it matches the library on simultaneous-noise mixtures, the hierarchy is not earning its keep, and the claim shrinks to an efficiency result.
Why not just train a deep enhancement network on lots of data? On in-distribution data it will probably win. Interactive Speech and Noise Modeling for Speech Enhancement reports 19.52 dB SDR on DNS, a level this proposal does not claim to reach. The claim here concerns noise combinations absent from training, and adding a new source without retraining anything else. The held-out-combination test below is designed so the deep reference is evaluated on combinations it never saw, which is the regime where explicit composition should pay off.
Proposed Experiments
The first decisive check measures interface loss. On TIMIT plus NOISEX-92 two-source mixtures at 0, 5 and 10 dB, it compares PESQ and STOI for the component interface against exact joint inference using identical components, with 8 to 40 speech states and 2 to 8 noise states.
The second check is the source-count sweep. Speech is mixed with one, two, three and four simultaneous NOISEX/MS-SNSD noises. Quality and wall-clock time are reported against the pruned joint model and the flat pooled model. The prediction is that the interface's time grows roughly linearly with source count, because cost is proportional to the sum of state counts, while its quality advantage over the pooled model grows with the number of overlapping noises.
The third check tests held-out combinations. All components and the deep reference are trained on single-noise mixtures only, then tested on pairs never seen together.
The fourth check is the nesting ablation. A "scene" composite with a gating chain is compared with the same children placed flat, on recordings constructed with slow scene switches.
The fifth check concerns lag. Fixed-lag smoothing at lag $L$ is compared with filtering alone on the van Erp and de Vries setup, plotting SNR gain against $L$. This tests whether the smoothing argument from Minami et al. carries over to measurable gains.
The last check is an ASR evaluation on CHiME at −6 dB through −6 to 9 dB, with recognition accuracy compared against the 75.42% S-Word result of Ma and Barker. It is a secondary check that the gains survive into recognition.
The proposal is refuted if either of two results appears. The first is that the interface loses more than about 0.2 PESQ to exact inference on two sources. The second is that the flat pooled noise HMM matches it on simultaneous and held-out noise combinations.
References
- An NMF-HMM Speech Enhancement Method Based on Kullback-Leibler Divergence
- Model-based noise suppression using unsupervised estimation of hidden Markov model for non-stationary noise
- Infinite Hierarchical Hidden Markov Models
- Coupling identification and reconstruction of missing features for noise-robust automatic speech recognition
- Online Single-Microphone Source Separation using Non-Linear Autoregressive Models
- GraphPPL.jl: A Probabilistic Programming Language for Graphical Models
- A theoretical analysis of speech recognition based on feature trajectory models
- Slice-and-Pack: Tailoring Deep Models for Customized Requirements
- Incorporating Symbolic Sequential Modeling for Speech Enhancement
- Interactive Speech and Noise Modeling for Speech Enhancement