Many tools that compare or classify speech, such as support vector machines, clustering, and fast nearest-neighbour search, need every input to be a vector of the same size. Spoken words and phrases are not like that: they vary in length and are stretched unevenly in time. Two families of methods handle this. Neural sequence encoders learn a fixed-size summary, but they need hundreds of hours of labelled pairs and struggle on rare words. Statistical summaries built from a Gaussian mixture model (a soft clustering of acoustic frames into a few hundred shared "sound prototypes") need little data, but they average over time and so lose word order. This note proposes a middle path drawn from a 1990s idea, the semi-continuous hidden Markov model. In that model, a small chain of states, each corresponding to a position inside a segment, all share one codebook of Gaussians, and each state differs from the others only in how heavily it uses each Gaussian.
The core idea fits in two sentences. Fit one small left-to-right semi-continuous model to untranscribed audio, softly align any segment to its $P$ positional states, and describe the segment by how far each position's use of the shared Gaussians departs from that position's typical use. This is the mixture-weight block of the model's Fisher score, the gradient of the log-likelihood with respect to the model's parameters. It is a fixed $P \times K$ vector whatever the segment's length, it keeps word order, it is learned almost entirely without labels, and it should hold up on rare words because every word reuses the same well-estimated codebook.
Core Claim
The claim is that tying parameters across states is what makes a sequence-model Fisher score usable as a segment embedding. Exploiting Generative Models in Discriminative Classifiers introduced the Fisher score $U_X = \nabla_\theta \log P(X\mid\theta)$, which has a fixed size for any sequence, and tested it with discrete hidden Markov models on DNA and protein data. For a continuous-density hidden Markov model, the score's size grows with states × mixtures × feature dimension, and most of those entries are poorly estimated. The tied-mixture model of Improved Hidden Markov Modeling for Speaker-Independent Continuous Speech Recognition and On The Use Of Tied-Mixture Distributions writes the emission density of state $j$ as $b_j(x)=\sum_k w_{jk}\,\mathcal N(x;\mu_k,\Sigma_k)$, so the Gaussian means and covariances are shared by all states.
That sharing splits the gradient into two parts. The first is the gradient with respect to the shared codebook. It depends only on component posteriors summed over states, so it is close to an order-blind Gaussian-mixture Fisher vector. All of the order information sits in the second part, the gradient with respect to the weights. Write each state's weights as a softmax, $w_{jk}=\mathrm{softmax}_k(\alpha_{jk})$. The occupancy-normalised weight score is then
$$ \phi_{jk}(X)=\frac{\sum_{t=1}^{T}\gamma_t(j,k)}{\sum_{t=1}^{T}\gamma_t(j)}-w_{jk}, $$
where $\gamma_t(j,k)$ is the forward-backward posterior that frame $t$ is in state $j$ and drawn from component $k$, and $\gamma_t(j)=\sum_k\gamma_t(j,k)$. In words, $\phi_{jk}$ is the extra share of Gaussian $k$ at position $j$ in this segment, compared with that position's prior share. Normalising by occupancy removes the dependence on length. With $P$ positional states shared across all words, the embedding has size $P\cdot K$ and does not grow with vocabulary size.
The closest existing representation is the probabilistic sequence kernel of A GMM-based probabilistic sequence kernel for speaker verification. It averages Gaussian posteriors over the whole utterance into a 1024-dimensional vector. That works for speaker identity, where order is deliberately ignored. Our representation is that kernel computed separately at each aligned position. The file's own tension between frame averaging, which Audio Word2Vec found clearly worse for spoken terms, and order-aware recurrent encoders is exactly the gap a positional alignment is meant to close.
Research Program
The model is a single left-to-right chain of $P$ states with self-loops, with $P$ between 4 and 12. Every state is required to be visited, so a segment must contain at least $P$ frames. The chain shares a codebook of $K$ diagonal-covariance Gaussians, with $K$ from 128 to 512.
Training draws on two prior papers. The first lesson comes from On The Use Of Tied-Mixture Distributions, which reports that plain k-means is a poor starting point for a tied-mixture codebook and that seeds should span phonetic space. We therefore initialise the codebook from a mixture trained on all frames and then split it. The chain is then trained by Baum-Welch on unlabelled segments: automatically detected word-like or syllable-like spans in the main experiments, with gold word spans as an upper-bound condition. The second lesson comes from the segment-region view in that paper's tied-mixture segment model, where a variable-length phone is mapped onto a fixed sequence of regions that share Gaussians. That paper used this structure only inside a recognition decoder. We reuse it to extract embeddings.
Given a segment, the program runs forward-backward over the chain, which costs time linear in $T \cdot P \cdot K$. It computes $\phi$, applies a square-root and power normalisation, and optionally appends the pooled codebook-mean gradient as a second block, which is tested in an ablation. The resulting vector can feed cosine retrieval directly. Where a small budget of labelled same-word pairs exists, it can instead feed a linear discriminant projection.
Three design questions are studied on their own. The first is the choice of $P$: too few states and order is lost, too many and states become sparse and the vector very long. The second is whether the Fisher-information whitening suggested by the original Fisher-kernel paper matters for the weight block. The third is input features. The primary features are MFCCs. As a secondary condition we use the DNN posterior features of CNN Based Query by Example Spoken Term Detection. Here A Comparative Evaluation of GMM-Free State Tying Methods for ASR acts as a warning: on WSJ eval92 it found that Gaussian likelihood-based tying fits neural outputs poorly, with 6.47% word error rate compared with 5.72% for a KL-divergence criterion. When neural features are used, we therefore also try codebook estimation based on KL divergence.
Baselines
The naive reference is the order-blind posterior average of A GMM-based probabilistic sequence kernel for speaker verification, together with its Fisher-vector variant over the same $K$-component codebook. It is computed from exactly the codebook our method uses. That makes it the cleanest test of whether positional structure adds anything. On its home task it reached 8.10% equal error rate on NIST SRE 2001. To beat it here, our method must show a clear gain in word-discrimination average precision under identical codebook, features and normalisation.
The ablation that isolates the key ingredient is uniform chunking. Each segment is cut into $P$ equal-length pieces, Gaussian posteriors are averaged within each piece, and the result is a vector of the same $P\cdot K$ size. This keeps everything except the learned soft alignment. If it matches our method, the hidden Markov model contributes nothing, and the claim reduces to "posteriorgrams downsampled to a fixed length." Our method beats it only if it wins overall, and especially on the speed-perturbation test described below. A second ablation removes the weight block and keeps only the codebook gradient, which shows directly that order lives in the weights.
The strongest existing method is the multi-view recurrent acoustic word embedding of Multi-view Recurrent Neural Acoustic Word Embeddings. It trains bidirectional LSTMs on audio and character sequences with a margin that grows with spelling edit distance. It reports word-discrimination average precision of 0.806 on Switchboard, against 0.671 for Siamese LSTMs and 0.549 for Siamese CNNs. We re-run it with the authors' recipe on each labelled-data budget, giving it the same tuning budget as our linear discriminant stage measured in configurations tried. We do not expect to beat 0.806 with full supervision.
The claim is about data efficiency. Our method wins if it equals or exceeds the multi-view model's average precision at small labelled budgets and on words seen fewer than a handful of times in training. To our knowledge Lacuna has no reported results for either baseline at reduced budgets, so those curves will be produced here. For query-by-example search, the reference systems are DTW and the CNN on similarity images of CNN Based Query by Example Spoken Term Detection. That CNN reports minimum normalised cross entropy of 0.6483 (lower is better), compared with 0.7181 for DTW, and maximum term-weighted value of 0.3753 compared with 0.3352 on SWS 2013.
Why not just…?
Why not just average Gaussian posteriors over the whole segment? Averaging throws away order, and order is what separates many word pairs, for example words made of the same sounds in a different sequence. Audio Word2Vec found frame averaging clearly worse than order-aware encoders for spoken term retrieval. The order-blind baseline above measures that gap directly on our codebook.
Why not just cut the segment into equal pieces and average within each? This is the real threat. It is cheaper, needs no forward-backward pass, and may capture most of the order information. Its weakness is that speech stretches unevenly: vowels lengthen with speaking rate while stops barely change, so fixed boundaries drift across sounds. If this ablation matches our method on clean data and under non-uniform time stretching, we will drop the hidden Markov alignment and report the simpler method. The proposal is built so that this outcome is a clean, publishable answer rather than a failure to hide.
Why not just train a neural acoustic word embedding? With abundant labelled pairs it will probably win, given its reported 0.806 average precision. Whole-Word Segmental Speech Recognition with Acoustic Word Embeddings reports 10.9% word error rate on the Switchboard 10k-word vocabulary, but its authors note that whole-word embeddings degrade on rare words and needed help from grounded written-word embeddings. Our claim covers only the low-label, rare-word regime, and the data-budget curves will show whether such a regime exists and where its boundary falls.
Why not just use DTW? DTW needs no fixed-length vector, but it costs quadratic time per comparison. Similarity Preserving Representation Learning for Time Series Clustering reports 169 minutes for DTW k-means on 16,637 sequences, against under 2 minutes for fixed-length embeddings. It also lost to a learned matcher on SWS 2013. We include DTW for accuracy and timing, not as the target to beat.
Proposed Experiments
The main experiment is the Switchboard same-different word discrimination task with the evaluation protocol of the multi-view paper. Average precision is reported overall and split by how often each word appears in training. The labelled-pair budget is swept from zero, using unsupervised embeddings with cosine similarity, up to the full set. All systems share the same untranscribed audio for unsupervised training.
The decisive check is speed perturbation. Evaluation segments are stretched both uniformly, at factors from 0.8 to 1.2, and non-uniformly by lengthening only frames in the high-energy vocalic middle. Average precision is then compared for the positional tied-mixture embedding and for uniform chunking. The hypothesis predicts that the two degrade similarly under uniform stretching but uniform chunking degrades clearly more under non-uniform stretching. If they degrade equally under both, the hypothesis is refuted.
The second experiment is query-by-example search on SWS 2013 and QUESST 2014. Search uses cosine scoring of query embeddings against sliding candidate windows, a window detail that is part of the protocol to be fixed. We report minimum normalised cross entropy, maximum term-weighted value and search time, compared with DTW and the reported CNN figures. For QUESST no reported result was found on Lacuna, so the baselines will be re-run.
A final sweep over $P$, over $K$, and over MFCC versus posterior features, with Gaussian versus KL-based codebooks, will show whether the mismatch reported in A Comparative Evaluation of GMM-Free State Tying Methods for ASR also limits embedding quality when the representation is built on neural features.
References
- Exploiting Generative Models in Discriminative Classifiers
- Improved Hidden Markov Modeling for Speaker-Independent Continuous Speech Recognition
- On The Use Of Tied-Mixture Distributions
- A GMM-based probabilistic sequence kernel for speaker verification
- Audio Word2Vec: Unsupervised Learning of Audio Segment Representations Using Sequence-to-Sequence Autoencoder
- CNN Based Query by Example Spoken Term Detection
- A Comparative Evaluation of GMM-Free State Tying Methods for ASR
- Multi-view Recurrent Neural Acoustic Word Embeddings
- Whole-Word Segmental Speech Recognition with Acoustic Word Embeddings
- Similarity Preserving Representation Learning for Time Series Clustering