Gaussian-Process Scheduling of Gaussian Evaluations in Speech Decoders

Classical speech recognizers and speaker-verification systems score every 10 ms audio frame against thousands of Gaussian components, the bell-shaped density pieces that make up a Gaussian mixture model. In large-vocabulary recognition this scoring takes close to 80% of decoding time (Chan et al.), and it overwhelms phone-class processors (Zhou et al.). The standard fixes are hand-tuned. A coarse model guesses which fine components matter. The previous frame's winner is reused. A hard cap limits how many components are scored. All three fail in the same place. Chan et al. report that the winning Gaussian repeats across frames about 70% of the time, but much less often for the highest-scoring components, which change fastest at stop bursts and fricative onsets (On improvements to CI-based GMM selection). Reuse heuristics therefore break exactly where accuracy matters most.

We propose to treat the scores of all components over time as an unknown function, and to place a Gaussian process (GP) prior on it. A GP is a Bayesian model over functions that returns a prediction together with a calibrated uncertainty. Each frame, the decoder scores exactly only the few components whose predicted score could plausibly enter the set of top candidates, and it lets the GP fill in the rest. The GP is not the acoustic model. It is a cheap scheduler that decides where a fixed per-frame budget of exact evaluations goes. Its uncertainty grows on its own when the audio changes abruptly, so it spends more evaluations at transients and fewer in steady vowels. Because the budget is fixed, decoding time per frame is constant.

Core Claim

The coarse-to-fine screening and frame-to-frame reuse tricks from the speech literature are crude, uncertainty-free special cases of inferring unobserved scores from observed ones. Making that inference explicit should improve matched-accuracy speed and worst-case latency. It requires a GP whose per-frame update costs far less than a full scoring pass, and that cost constraint is where GP scalability work becomes the core mechanism.

Let $s_{k,t} = \log w_k \mathcal{N}(x_t;\mu_k,\Sigma_k)$ be the log-score of component $k$ (with weight $w_k$, mean $\mu_k$, covariance $\Sigma_k$) on frame $x_t$. We model $s$ with a covariance that factors into two parts. The first is a component part, defined by a tree obtained by merging components in order of Kullback–Leibler divergence. The second is a temporal part, a Markov (Ornstein–Uhlenbeck) process whose step variance is scaled by how much the features moved:

$$ \mathrm{Var}(s_{k,t}-s_{k,t-1}\mid \text{past}) \;=\; q_k\,\lVert x_t-x_{t-1}\rVert^2 + r_k , $$

where $q_k$ and $r_k$ are per-cluster constants fitted offline. The norm $\lVert x_t-x_{t-1}\rVert$ is shared across all components and costs one 39-dimensional difference per frame. The scheduler then repeatedly picks components with the highest posterior probability of clearing the current cutoff $\theta_t$, defined as the $N$-th best exactly computed score or the beam threshold:

$$ a_{k,t} \;=\; \Pr\!\left(s_{k,t}\ge\theta_t\right) \;=\; \Phi\!\left(\frac{m_{k,t}-\theta_t}{\sigma_{k,t}}\right), $$

where $m_{k,t}$ and $\sigma_{k,t}$ are the GP posterior mean and standard deviation of the score and $\Phi$ is the standard normal distribution function. Picking stops at a fixed budget $B$.

The tree makes the within-frame posterior exact and computable in one upward and one downward message pass. A subtree whose root has an upper bound $m+3\sigma$ below $\theta_t$ is never expanded, so most leaves are touched only through their cluster ancestor. Across time we keep only per-node marginals. This is a Kalman-style filter that drops cross-time correlations between nodes, and it keeps memory and per-frame time constant.

Relation to Prior Work

Prior work in three roles motivates and constrains this design. The motivating evidence comes from Chan et al.. Their context-independent pre-screening, best-Gaussian-index reuse and absolute caps reached a 37% speedup with about 3% relative word-error increase, and their locality paradox is the failure mode we target. The clustered background model of Liu et al. supplies our tree construction (KL-divergence merging) and our speaker-verification testbed. Its top-$N$ cluster gating is a hard, uncertainty-free version of our subtree expansion.

On the GP side, Approximation-Aware Bayesian Optimization shows that a sparse GP should spend its limited budget on decision-relevant regions rather than on global fit. We apply the same principle, but to exact-evaluation budget rather than inducing points, and with a decoding cutoff as the utility. The constant-time recursive summaries of GP-Localize and the ODE-updated memory of OHSVGP supply the template for a GP whose per-step cost does not grow with stream length.

Several existing works combine GPs with Markov or mixture structure, and all of them use the GP as the observation or regression model, which is a different job from ours. HM-GPFR puts GPs inside HMM states for scalar load curves. Variational Mixture of GP Experts gates GP experts with a memoryless mixture model. ICDKGP ties inducing points to data clusters. We leave the trained Gaussian mixtures untouched and use the GP only to decide which of them to compute, so recognition accuracy can at worst match the unpruned system.

Fabian et al. use confidence to narrow the search over word hypotheses. That method is complementary: it prunes paths, while we schedule likelihood evaluations within surviving paths. Finally, Sparse GP Hyperparameters: Optimize or Integrate? warns that maximum-likelihood fitting underestimates uncertainty. Our scheduler depends on calibrated $\sigma$, so we fit only a handful of per-cluster constants and check calibration directly.

Research Program

We will build the scheduler inside Sphinx-3, whose four-level evaluation structure (time, mixture, Gaussian, component) already isolates the scoring loop that Chan et al. modified. We will also build it inside a factor-analysis speaker-verification pipeline that follows Liu et al.

Offline, we build the component tree. We then fit $q_k$, $r_k$ and the tree's edge variances by maximizing the likelihood of exact score trajectories on held-out, force-aligned training audio. We check calibration of the predictive intervals separately at phone boundaries and in phone interiors.

Online, each frame runs in four steps. First, propagate the marginals one step. Second, run the bounded subtree expansion. Third, compute $B$ exact scores in a few small batches, re-running the message pass between batches. Fourth, use exact scores where available and posterior means elsewhere. Using posterior means plays the role that back-off to coarse scores plays in Chan et al.

The analysis has three parts. We measure the overhead of the scheduler in floating-point operations and wall-clock time relative to one diagonal-Gaussian evaluation. We measure how the realized budget allocation shifts toward transients. We characterize the regime, in mixture count and feature dimension, where the scheduler pays for itself. A predefined kill criterion applies: if bookkeeping costs more than about a third of a full scoring pass on a single CPU core, the approach is not viable on embedded hardware, and we report that result.

Baselines

The naive reference is exhaustive scoring of every Gaussian every frame with the same models and features. On Communicator, Chan et al. report this configuration at 0.89xRT (real-time factor). It defines the accuracy ceiling, which our method should match at large $B$.

The strongest existing method on this exact question is the combined absolute pruning, best-Gaussian-index and adaptive context-independent selection system of Chan et al.. It reaches 0.56xRT on Communicator at roughly a 3% relative word-error increase. It is the strongest competitor because it already exploits both coarse screening and temporal reuse, the two kinds of structure our GP models. We will reimplement it in the same Sphinx-3 build and give its beam, cap and frame-downsampling thresholds the same grid-search budget on the same development set that our $B$ and calibration constants receive. Beating it means a lower real-time factor at matched word error rate on Communicator, WSJ-5k and ICSI meetings, together with a lower 99th-percentile per-frame latency. For WSJ-5k and ICSI, no reported absolute word-error or real-time numbers were found on Lacuna, so both systems will be measured fresh.

For speaker verification, the baseline is the clustered background model with top-3 selection over 16 clusters from Liu et al.. They report 1.24 s and 3.89% equal error rate on NIST SRE, against 9.53 s and 4.25% for full factor analysis. We run it on the same SRE trial lists. Beating it means equal or lower equal error rate and minimum detection cost at lower time, plus smaller error in the accumulated sufficient statistics.

Ablations will isolate the two ingredients. A temporal-only GP drops the tree, leaving an independent per-component filter. A tree-only GP drops time and is a probabilistic cousin of context-independent pre-screening. A mean-only scheduler ranks by $m_{k,t}$ and ignores $\sigma_{k,t}$. The mean-only ablation tests whether uncertainty, rather than better point prediction, drives the gain.

Why not just…?

Why not just tune the existing heuristics harder? Chan et al. themselves show that gains from stacking heuristics shrink to 5–10% per technique on WSJ-5k. Their failure is structural: reuse is wrong at fast-changing high scorers, and no fixed threshold knows when a transient is happening. Our experiment settles the question by breaking errors down by distance to the nearest forced-aligned phone boundary. If the tuned heuristics show no excess top-1 misses near boundaries, the paradox is not a practical problem and our motivation weakens.

Why not just use a deterministic bound and skip components that provably cannot win? This is the most serious threat. The score change between frames can be bounded using the stored Mahalanobis distance, the largest precision and $\lVert x_t-x_{t-1}\rVert$, all at constant cost per component, which gives certified skipping with no learning. We expect this worst-case bound to be loose in 39 dimensions, but that is a hypothesis. We will implement it as a full baseline. If it matches the GP at equal accuracy, we will report that the simple certificate suffices and that the GP's value is limited to latency smoothing.

Why not just evaluate everything on a GPU? Recasting scoring as dense matrix multiplication gives about an 11.6x speedup (Accelerate Acoustic Likelihood Computations on GPU). Where a GPU exists, exhaustive scoring may simply win. Our target regime is the CPU-only phone and embedded setting described by Zhou et al.. We will state plainly that the method is not intended for GPU servers.

Why not just replace Gaussian mixtures with a neural acoustic model? Gupta & Boulianne find that neural acoustic models carry their own heavy compute, so the cost problem moves rather than disappears. Gaussian-mixture posteriors also remain central to factor-analysis and i-vector speaker systems (Dehak et al.). We do not claim to beat neural recognizers. The claim concerns the mixture-based pipelines that remain in use.

Proposed Experiments

On Communicator, WSJ-5k and ICSI, we will trace real-time factor against word error rate for every system by sweeping $B$ or the baseline thresholds. We will report latency percentiles per frame and the top-$N$ recall of exactly computed components against exhaustive scoring, broken down by distance to the nearest phone boundary. On NIST SRE 2004–2006 we will report equal error rate, minimum detection cost, extraction time and relative error of the zeroth- and first-order statistics.

Three checks are decisive. The first is whether, at the word error rate of Chan et al.'s best configuration, the scheduler runs faster than 0.56xRT on Communicator with a flatter latency tail. The second is whether the mean-only ablation loses most of that gain, which would show that calibrated uncertainty, not prediction, is the ingredient. The third is whether the deterministic-bound baseline falls clearly short. A positive result on all three would show that GP inference can replace hand-tuned selection heuristics in mixture-based decoders. A loss to the deterministic bound would be a clean negative result worth publishing.