Are "Lossless" Speech Recognition Speedups Lossless for Everyone?

Fast speech recognizers on phones and commodity CPUs depend on approximations. They compress the acoustic model's Gaussian distributions into small lookup tables, run the scoring in low-precision vector arithmetic, and prune the search aggressively. Papers report these tricks as costing nothing because average word error rate (WER) does not move. Datasets, however, are not uniform. The speech-dataset documentation literature shows that corpora mix accents, dialects, microphones and noise conditions that a single average hides. We propose to test whether the approximation error introduced by fast likelihood kernels concentrates on underrepresented speakers and recording conditions. We also propose a cheap, transcript-free way to see this concentration in advance: compare the approximate and exact per-frame scores on audio grouped by the strata a datasheet documents. If the concentration is real, the same documentation can be used to build quantization tables that do not shortchange those groups.

The opportunity is that two literatures have each stopped one step short. Work on accelerated likelihood computation measures accuracy only on aggregate benchmarks. Work on dataset transparency describes subpopulations carefully but never asks how the deployed scoring engine treats them. Joining them yields a concrete artifact, an "acceleration card" that reports per-stratum cost for each speedup. It also yields a concrete method: quantization codebooks trained with stratum weights.

Core Claim

A likelihood kernel is the inner loop that computes, for every audio frame (about 10 ms of sound), how well each Gaussian mixture in a hidden Markov model (HMM) acoustic model explains that frame. Towards High Performance LVCSR in Speech-to-Speech Translation System on Smart Phones made this kernel fast on a 1 GHz phone in three ways. It used ARM NEON vector instructions (SIMD, single-instruction-multiple-data). It quantized the Gaussians into sub-bands with lookup tables. It compressed the feature transform by k-means clustering. It reports English WER of 18.8% both before and after quantization. Parallel fast likelihood computation for LVCSR using mixture decomposition split the same computation across cores and reports "no degradation in WER" on EPPS English. Both papers motivate our question, and both judge success by a corpus-wide average.

Our hypothesis is mechanistic. The codebooks behind sub-band quantization are fit by k-means on pooled training data, so they spend their resolution where the data is dense. Speech from accents or channels that are rare in training falls into regions of feature space that the codebook covers coarsely. As a result, the per-frame score error

$$ \epsilon_t = \left|\log p_{\text{exact}}(x_t \mid s) - \log p_{\text{approx}}(x_t \mid s)\right|, $$

where $x_t$ is the feature vector at frame $t$ and $s$ is an HMM state, should be systematically larger for minority strata. That larger error should then become larger WER increases once beam pruning removes the correct path. Measuring $\epsilon_t$ needs no transcripts and no decoding. It therefore scales to unlabeled audio, and it can be reported per stratum alongside the documentation that defines the strata.

The proposal does three new things. It evaluates speedups across documented strata, which no paper in the file does; Gap 3 of the file names this as open. It offers per-frame score error as a cheap predictor of where WER damage will appear. And it introduces stratum-weighted codebook training as a fix that keeps speed fixed.

Research Program

We first build a reference exact kernel: float32 diagonal-covariance Gaussian scoring using AVX2 and NEON. Next to it we build three approximate kernels at matched real-time factor. The first follows Zhou et al.: sub-band quantization with lookup tables, plus fixed-point arithmetic. The second is the exact kernel with a tighter beam. The third is the exact kernel run with Parihar et al.'s chunk-based and decision-tree mixture decomposition, which reorders work but should change no scores. That third kernel serves as a control. All kernels run against a single GMM-HMM acoustic model trained on LibriSpeech 960h.

Strata come from datasheets rather than from our own guesses. Augmented Datasheets for Speech Datasets and Ethical Decision-Making provides reference datasheets for LibriSpeech, Common Voice, VoxPopuli, WHAM and CORAAL. These document accent, dialect, recording conditions and noise. We operationalize their fields as evaluation groups. For Common Voice these are self-reported accent and age. For CORAAL they are speaker community and recording era. For channel effects we use AMI close-talk versus single distant microphone, and CHiME-6. The datasheets play a constraining role: they fix the stratification before we see any results, which guards against slicing the data until a gap appears.

The fix is stratum-weighted codebook training. When fitting the sub-band k-means, each frame is weighted inversely to its stratum's share of the training data. The weights use datasheet metadata where it exists. Where it does not, they use an unsupervised proxy: the frame's $\epsilon_t$ under the pooled codebook. The lookup tables keep the same size and memory budget (under 150 MB total, following Zhou et al.'s device constraint), so speed is unchanged by construction.

Baselines

The naive reference point is the exact float32 SIMD kernel with a standard beam. It defines zero approximation error, and every other system's per-stratum WER is reported as a difference from it. This makes model quality and approximation cost separable.

The strongest existing method is Zhou et al.'s quantized lookup-table GMM, reimplemented with pooled k-means codebooks at its reported compression level. It is the strongest competitor because it is the only approach in the file that pairs large memory reduction with a reported zero average loss. The cited figure is English WER 18.8% quantized versus 18.8% unquantized on DARPA Transtac, from Towards High Performance LVCSR in Speech-to-Speech Translation System on Smart Phones. We cannot use Transtac, so the comparison will be made on our own acoustic model, decode settings and real-time factor. The file contains no reported subgroup result: no reported result was found on Lacuna for stratified WER of quantized or pruned GMM kernels on CORAAL or Common Voice accent subsets.

The pruning baseline is the exact kernel with its beam narrowed until it reaches the quantized kernel's speed. This tests whether score approximation or search approximation is responsible for any disparity.

Parihar et al.'s mixture decomposition serves as a sanity control. Its WER should be unchanged in every stratum, because it computes identical scores.

Two ablations isolate the key ingredient. The first trains the weighted codebook with stratum labels randomly permuted, which keeps the weighting mechanics but removes the information. The second uses only the unsupervised $\epsilon_t$ weighting, which asks whether documentation adds anything.

All codebooks get the same table budget and the same number of k-means restarts, and are tuned on a shared development split stratified identically. Beating the Zhou-style baseline means two things. At matched real-time factor and memory, the stratum-weighted codebook must reduce the worst-stratum WER increase over exact scoring by a margin outside bootstrap confidence intervals over speakers. It must also not raise the average WER increase.

Why not just…?

Why not just report disaggregated WER for existing fast decoders? That is our first experiment, and if it shows no disparity the fix is moot; we would report that as a negative result. Disaggregated WER alone has two limits. It needs transcribed audio for every stratum, and it cannot separate a weak acoustic model from a lossy kernel. Measuring the difference from the exact kernel together with the transcript-free $\epsilon_t$ addresses both limits, and we test directly whether $\epsilon_t$ predicts the WER differences.

Why not just retrain the acoustic model on balanced data? Rebalancing changes the model and confounds the question we are asking. It is also more expensive than refitting a lookup table. We hold the model fixed so that any disparity can be attributed to the approximation. Rebalanced training is complementary, and we include one rebalanced model to check whether kernel disparity persists when the model itself is fairer.

Why not just use the unsupervised error weighting and skip the datasheets? This alternative is a real threat. If error-weighted k-means matches metadata weighting, documentation is unnecessary for the fix, and its role shrinks to defining the evaluation strata. The second ablation settles this directly. We would accept either outcome; a tie is a cheaper method that works for undocumented corpora.

Why not just run exact scoring on a GPU? The approximations exist because deployment targets such as phones and browsers lack GPUs and have tight memory budgets, as Zhou et al.'s 512 MB device and Exploring web-browser based runtimes engines for creating ubiquitous speech interfaces illustrate. The question concerns the systems people actually ship.

Proposed Experiments

The first decisive check is whether disparity exists at all. On Common Voice accent subsets, CORAAL, and AMI close-talk versus distant microphone, we compute each stratum's WER increase under the quantized and narrow-beam kernels relative to exact scoring. The hypothesis survives only if the increase varies across strata beyond speaker-level bootstrap noise, and only if the mixture-decomposition control shows no such variation.

The second check is whether the cheap signal predicts the damage. We measure the rank correlation, across strata and across utterances, between mean $\epsilon_t$ and the WER increase. A useful predictor would let a team audit a new corpus with one fast pass of both kernels and no transcripts.

The third check is the fix. We compare stratum-weighted, error-weighted and permuted-label codebooks against the pooled Zhou-style codebook at matched memory and real-time factor. We report the worst-stratum increase, the average increase, and throughput in frames per second on both AVX2 and NEON hardware.

The final deliverable is a machine-readable acceleration card for each kernel. It lists per-stratum WER increases and $\epsilon_t$ distributions, keyed to the datasheet fields that define the strata. The format follows the schema approach of Open Datasheets, so a claim that a speedup is lossless becomes something a reader can check against the subgroups it covers.