Self-supervised speech models such as HuBERT learn by hiding stretches of audio and predicting a discrete label for each hidden frame. Nobody hands them those labels, so HuBERT manufactures them in rounds. It first clusters simple acoustic features (MFCCs, a standard hand-designed spectral summary) with k-means and trains a Transformer to predict those clusters. It then re-clusters that Transformer's own internal features and trains a new Transformer from scratch on the better labels. Every round repeats a full pretraining run, so most of the cost buys labels rather than the final model. We propose to produce labels good enough for the final round directly from MFCCs, in one cheap pass and without a neural network. The labels come from a subspace Gaussian mixture model (SGMM): a classical speech model in which thousands of sound states share one pool of full-covariance Gaussians and differ only by a short coordinate vector. The Transformer is then pretrained once.
The bet rests on two findings. First, PolyBERT showed that one round of pretraining beats iterated HuBERT when the targets are context-dependent phonetic units, but it obtained those units from an adversarial text-matching pipeline. Second, SGMMs are known to support very large, well-regularized state inventories from little data and to model correlated features that diagonal Gaussians miss. If an SGMM-HMM fitted to unlabeled MFCCs can supply PolyBERT-like targets, the second pretraining round and the adversarial stage both disappear. In two sentences: HuBERT pays for extra full training runs only to improve its targets, and subspace Gaussian HMMs are a cheap, stable, non-neural way to get fine-grained, temporally smoothed targets up front. We test whether one pretraining pass on such targets matches multi-round HuBERT at a fraction of the total compute.
Core Claim
HuBERT trains a Base model on k-means(100) clusters of MFCCs. It then extracts layer-6 features over all 960 hours of LibriSpeech, runs k-means(500), and retrains from scratch. Its authors name a single-phase procedure and better clustering as open directions. Pushing the Limits of Unsupervised Unit Discovery for SSL Speech Representation answers half of that. Its PolyBERT-PT trains a 95M-parameter model once, on triphone-like units derived from a modified wav2vec-U 2.0, and reports 11.1% test-other word error rate (WER) after 100h fine-tuning without a language model. Iterated HuBERT Base reaches 13.2% in the same setup. The unit source, however, is a generative adversarial pipeline trained against unpaired text, which is multi-stage and unstable.
Our claim is that a subspace Gaussian HMM fitted without labels to MFCCs can replace that pipeline. An HMM here is a model that moves between states frame by frame, and each state emits frames through its own mixture of Gaussians. In the SGMM version, the state means are $\mu_{mi} = M_i v_m$, where $M_i$ is a projection matrix shared by all states for pool component $i$ and $v_m$ is a low-dimensional vector belonging to state $m$. Full covariances $\Sigma_i$ are shared across states, as described in Application of Subspace Gaussian Mixture Models in Contrastive Acoustic Scenarios.
Three properties make this a plausible target generator. The HMM's state transitions smooth labels over time, which frame-wise k-means cannot do. Sharing everything except $v_m$ lets us afford thousands of context-sensitive states without each one overfitting. Shared full covariances capture correlations between feature dimensions. The approach is new in using the SGMM as the label source for masked-prediction pretraining, with total compute (label generation plus pretraining) as the primary axis of comparison.
Research Program
We build three target generators. Each produces frame-level discrete labels for LibriSpeech 960h.
The main one is an unsupervised SGMM-HMM. We fit a universal background model, a single Gaussian mixture of a few hundred components covering all MFCC frames. We then grow a flat inventory of discovered states, initialized from k-means segments, by splitting states and re-estimating with EM. Following the generalized subspace view in Bayesian Subspace Hidden Markov Model for Acoustic Unit Discovery, we expand selected states into context-dependent variants by clustering their occurrences on neighbouring-state identity. This imitates PolyBERT's physical triphones without any phone labels.
The second generator is the Bayesian subspace HMM of Ondel et al. It uses a phonetic subspace learned from labeled GlobalPhone languages (German, Polish, Spanish). That paper reports 49.2% equivalent phone error rate on TIMIT with a 30-dimensional subspace. Because this variant uses outside supervision, we report it separately and do not count it toward the purely self-supervised claim.
The third generator is a precision-constrained variant. Comparison of subspace methods for Gaussian mixture models in speech recognition found that tying only precision matrices to a subspace was stable. Tying all exponential-family parameters, as in its SCGMM variant, behaved erratically and overtrained at larger subspace dimensions. That result constrains our design. We therefore sweep subspace dimension and tying scope, and we monitor target quality as the state inventory grows.
All generators feed an unchanged HuBERT Base masked-prediction recipe for a single pretraining run. EM on Gaussians over 39-dimensional MFCCs runs on CPUs and is itself iterative. We therefore log its wall-clock time and energy alongside GPU-hours, so that the claim of "removed overhead" cannot hide a moved overhead.
Before any pretraining, we screen target quality cheaply. On TIMIT, we measure phone purity, phone-normalized mutual information against forced alignments, and equivalent phone error rate. This lets weak generators be discarded early.
Baselines
The naive reference point is a single round of HuBERT on k-means(100) MFCC targets, using the same architecture and the same number of updates as one round. It shows what one pass buys with no better targets. We also run it at a matched label inventory, k-means with as many clusters as our SGMM has states, so that inventory size alone is not credited to the subspace.
The strongest compute-heavy baseline is standard two-round HuBERT Base, for which the PolyBERT paper reports 13.2% test-other WER (100h fine-tuning, no language model). We rerun it ourselves on our hardware so that total GPU-hours are measured under identical conditions. No reported compute figure for HuBERT pretraining was found on Lacuna. The slimIPL G/TPU-day figures concern a different, semi-supervised pipeline and are not comparable.
The strongest single-pass method is PolyBERT-PT from Pushing the Limits of Unsupervised Unit Discovery for SSL Speech Representation, with 11.1% test-other WER in the same setup. We cite this number and, if their unit-discovery code is reproducible within budget, rerun it to obtain its target-generation cost.
The ablation that isolates the key ingredient is a diagonal-covariance GMM-HMM and a full-covariance GMM-HMM without a subspace. Both use the same state inventory, the same context expansion and the same EM budget. If either matches the SGMM, the gain comes from HMM smoothing or inventory design rather than subspace sharing.
All systems get the same pretraining data, update count per round, fine-tuning recipe and hyperparameter search budget for fine-tuning learning rate. Beating iterated HuBERT means at or below 13.2% test-other WER at no more than about 60% of its total compute. Approaching PolyBERT means landing within one point of 11.1% with no text and no adversarial stage.
Why not just…?
Why not just warm-start HuBERT's second round instead of retraining from scratch? This is the most serious threat to our efficiency argument, because it may recover much of the saved compute with no new modelling. It still requires a first pretraining run, a full-corpus feature dump and k-means. We include warm-started two-round HuBERT as an extra compute-matched baseline. If it reaches our WER at equal total compute, our contribution narrows to target quality per unit of compute, and we will say so.
Why not just use more diagonal Gaussians or k-means clusters? On bottleneck features, Subspace models for bottleneck features found that diagonal GMMs gained about 5% relative over raw features, while SGMMs gained 31.9% relative over the diagonal baseline. Varjokallio et al. found that doubling diagonal components cost 23–24% more parameters than the precision-constrained model without matching its accuracy. Those results come from supervised recognition, not label generation, so the matched-inventory k-means and diagonal GMM-HMM baselines are what settle the question here.
Why not just decorrelate with PCA and run k-means? The bottleneck-feature study found the Karhunen–Loève transform (a PCA-style decorrelation) gave only 6.3% average relative gain, with marginal benefit on top of SGMMs. A single global rotation cannot represent covariances that differ by region of acoustic space. We still run PCA-whitened k-means in the TIMIT screen because it costs almost nothing.
Why not just use full-covariance GMMs without a subspace? Varjokallio et al. report that full-covariance models overtrain at 16 Gaussians per state. A large discovered-state inventory with few frames per state is exactly that regime. The full-covariance ablation tests whether this failure appears for target generation as well.
Proposed Experiments
Stage one is the TIMIT target-quality screen across all generators, with the Bayesian subspace HMM's 49.2% equivalent phone error rate as the reference. Stage two pretrains HuBERT Base once on each surviving target set from LibriSpeech 960h. We then fine-tune on the 100h and 10h splits and report dev and test WER without a language model. The central figure plots test-other WER against total compute, with label generation included.
We also run SUPERB speaker identification. PolyBERT dropped to 77.52% against HuBERT's 81.42%, which suggests phone-like targets can strip speaker information. Our shared-pool-plus-coordinates design may show the same trade-off, and the result belongs in the report either way.
The decisive check has two parts. A single-pass SGMM-target model must match or beat rerun two-round HuBERT on test-other at clearly lower total compute. It must also beat the matched-inventory diagonal GMM-HMM, so that the subspace, rather than HMM smoothing alone, earns the credit. If the first part holds and the second fails, we report the cheaper finding, temporally smoothed Gaussian HMM targets, and drop the subspace claim.
References
- HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units
- Pushing the Limits of Unsupervised Unit Discovery for SSL Speech Representation
- Application of Subspace Gaussian Mixture Models in Contrastive Acoustic Scenarios
- Bayesian Subspace Hidden Markov Model for Acoustic Unit Discovery
- Comparison of subspace methods for Gaussian mixture models in speech recognition
- Subspace models for bottleneck features