Unsupervised speech recognition learns to transcribe speech with no paired audio and transcripts. A generator turns audio features into phoneme sequences, and a discriminator, trained on unrelated text, judges whether those sequences look like real language. A central weakness of these systems is how they decide where one phoneme ends and the next begins. Current systems handle this in one of two ways. Some compress audio at a fixed rate, which ignores stretched vowels and pauses. Others learn boundaries with reinforcement learning, which cannot pass gradients from the recognizer back into the boundary decisions.
We propose a third option: a segmentation layer that is differentiable and runs in linear time. The generator predicts a nonnegative "phoneme mass" for each audio frame. Running sums of that mass define a monotonic transport plan, which assigns each frame to one or more output slots in time order. Because the plan is the exact one-dimensional monotonic optimal transport between frame masses and uniform slot masses, it costs $O(T+K)$ and has gradients everywhere. The same plan is used twice. Run forward, it pools frames into phoneme slots for the discriminator. Run backward, it spreads the predicted phonemes back over the frames, so the model can be asked to reconstruct its own audio features. That reconstruction loss is an unsupervised, elastic stand-in for the alignment loss that supervised adversarial speech models found indispensable.
Core Claim
A single learned monotonic transport plan can do three jobs that are currently handled separately and less well. It can replace the fixed stride of wav2vec-U 2.0. It can replace the reinforcement-learned segmenter of REBORN. It can replace the MFCC-cluster grounding loss that wav2vec-U 2.0 needs to avoid mode collapse, where the generator ignores the audio and emits plausible text. All of this happens in one backward pass, at a cost linear in utterance length.
The claim rests on two pieces of prior evidence. First, trainable boundaries clearly help: REBORN reports 5.4% PER and 9.6% WER on LibriSpeech test-clean. Second, adversarial feedback alone does not learn alignment: End-to-end Adversarial Text-to-Speech (EATS), a supervised adversarial text-to-speech model, needed a soft-DTW reconstruction term to converge. No work in the file has built a differentiable, linear-cost segmenter inside an unpaired adversarial recognizer.
Mechanism
Let $h_1,\dots,h_T$ be frozen wav2vec 2.0 frame features. A small convolutional network predicts masses $\alpha_t = \mathrm{softplus}(f(h_t))$, and $c_t=\sum_{s\le t}\alpha_s$ is their running sum. The number of output slots is $K=\lceil c_T\rceil$. Frame $t$ contributes to slot $k$ with weight
$$w_{t,k} = \big|\,[c_{t-1},c_t) \cap [k-1,k)\,\big|,$$
which is the length of overlap between the frame's mass interval and the slot's unit interval. The slot representation is $z_k=\sum_t w_{t,k} h_t$. This is the northwest-corner monotonic transport plan used by OTTC, which introduced it as a linear-time alignment loss for supervised ASR. We reuse it as a learned pooling operator rather than as a loss.
The operator has three useful properties:
- Each frame touches only $\lceil\alpha_t\rceil+1$ slots, so time and memory are $O(T+K)$.
- A frame with $\alpha_t\approx 0$ effectively disappears, which is how OTTC removes blank and silence frames.
- Gradients reach every $\alpha_t$, through both the overlap weights and the fractional final slot.
A phoneme classifier maps each $z_k$ to a distribution $p_k$. The discriminator sees $p_{1:K}$ exactly as in wav2vec-U 2.0.
For grounding, we unpool through the transpose of the plan, $\hat h_t = g\big(\sum_k w_{t,k}\,p_k/\alpha_t,\ s\big)$. Here $s$ is a single utterance-level speaker vector, and the objective is to predict $h_t$ (or its quantized targets). The only per-frame information reaching the reconstructor passes through the phoneme posteriors. Mode collapse therefore becomes directly costly: outputs that ignore the audio cannot reconstruct it.
Two regularizers complete the objective:
- A weak prior pulls the average mass rate toward the 10 to 15 phonemes per second that the file cites as typical.
- A temperature schedule on $f$ gradually sharpens $\alpha$.
We deliberately allow slots finer than phonemes, with adjacent identical predictions merged. REBORN found that sub-phonemic segments help, and OTTC's strict one-label-per-slot alignment paid a small WER penalty (8.76% versus 8.38% on TIMIT).
Relation to Prior Work
Three lines of work supply the ingredients, and each stops short of this combination.
The first is soft alignment. Soft-DTW and its use in EATS give smooth alignment losses, but they need a paired reference sequence, which does not exist in unsupervised ASR, and they cost quadratic time and memory. Learnable Dynamic Temporal Pooling avoids the pairing problem by aligning to $L$ learned prototypes. However, it fixes $L$ in advance, whereas phoneme counts vary from utterance to utterance.
The second is parametric segmentation. Differentiable Segmentation of Sequences warps time with smooth distribution-based boundaries, but it was only demonstrated on a nine-boundary TIMIT example with a fixed number of change points.
The third is unsupervised ASR itself: wav2vec-U, wav2vec-U 2.0, and REBORN define the task. Their segmentation is either heuristic (clustering or a fixed stride) or non-differentiable (reinforcement learning with behavior-cloning initialization).
Our proposal differs from all three in the same way. The number and position of segments are free, learned from gradients of the discriminator and the reconstruction loss, and computed in linear time.
Research Program
We will build the mass-based pooling layer and the transpose-reconstruction loss on top of the wav2vec-U 2.0 codebase setup: the same layer-15 features of a large wav2vec 2.0 model, the same phonemized text, and the same discriminator.
The analysis asks three questions. First, where does the boundary signal come from? We measure the gradient norm reaching $\alpha$ from the discriminator versus from reconstruction. Second, do the learned masses line up with human phoneme boundaries? Third, does the linear-cost operator make training on full-length and long utterances practical where soft-DTW-style pooling runs out of memory?
Baselines
The naive reference point is wav2vec-U 2.0 with its fixed stride of 3 and MFCC grounding loss. It reports LibriSpeech dev-other PER of 13.6% and 6.3% WER after self-training. We rerun it with identical features, text, and training steps.
The strongest existing method is REBORN. It reports TIMIT PER of 12.4%, and LibriSpeech test-clean PER of 5.4% and WER of 9.6% after HMM self-training. We will run it from released settings on identical data and compare at matched stages, both before and after self-training, because refinement can hide differences in segmentation quality. We note that wav2vec-U reports TIMIT PER falling from 26.1 to 11.3. That figure is lower than REBORN's, which suggests the two papers compare at different pipeline stages. We will report both our own reruns and the published numbers, and label each.
Beating REBORN means matching or lowering its PER before self-training, with a single-stage training loop and no behavior-cloning initialization. Because no reported result was found on Lacuna for boundary F1 of unsupervised ASR segmenters on TIMIT, we will compute that metric for REBORN, wav2vec-U 2.0, and our model ourselves.
Two ablations isolate the key ingredient, and every variant gets the same tuning budget of an equal number of seeds and hyperparameter trials:
- We keep the transport pooling but drop reconstruction, which tests EATS's claim that adversarial signal alone is insufficient in the unpaired setting.
- We keep stride 3 but replace the MFCC loss with our reconstruction loss. This tells us whether the gain comes from grounding alone.
Why not just…?
Why not just keep REBORN's reinforcement-learned segmenter? It works, and it is the real bar to clear. Its costs are an alternating two-model loop, initialization from clustering boundaries, and no gradient flow from the recognizer into boundary placement. If our model only ties REBORN, the contribution shrinks to simplicity. We will report wall-clock time and variance across seeds, since policy-gradient training usually has high variance. A tie with lower seed variance and one training stage would still be a useful result, but we will state it as that.
Why not just use hard boundaries with a straight-through gradient estimator? This is the most serious threat, because it also gives end-to-end gradients cheaply. We will implement it inside the same pooling layer, thresholding $\alpha$ into hard cuts on the forward pass. If it matches our method on PER and boundary F1, then fractional transport is unnecessary and we will say so. We expect it to struggle early in training, when boundaries are uninformative and biased gradients push segments toward degenerate lengths. The temperature-schedule comparison tests this directly.
Why not just fix grounding and keep the stride? Mode collapse, rather than segmentation, may be the real bottleneck. The stride-3-plus-reconstruction ablation answers this question directly. If it recovers most of the gain, the reconstruction loss is the finding, and adaptive segmentation is secondary.
Why not soft-DTW pooling with a Sakoe-Chiba band? Soft-DTW needs a target sequence or a fixed prototype count. A band assumes the alignment stays near the diagonal, which fails for pauses and changes in speaking rate. We will measure its memory use and PER on long LibriSpeech utterances as a complexity contrast.
Proposed Experiments
TIMIT gives both PER and boundary precision, recall, and F1 against human annotations. It is the primary test of whether the masses find phonemes or sub-phonemes. LibriSpeech provides dev-other and test-clean PER, and WER after HMM self-training. MLS tests generalization across its six languages, and the CommonVoice Tatar and Kyrgyz subsets test low-resource robustness; wav2vec-U 2.0 reports PER of 10.1% and 15.2% on these.
A stress test uses LJSpeech with characters instead of phonemes, where wav2vec-U 2.0 collapses from 14.4% to 64.0% WER. Adaptive, sub-phonemic slot rates could plausibly help here, because characters do not map one-to-one onto sounds. This is an exploratory test, not a core claim.
Three checks decide the proposal:
- Seed robustness: does transport pooling with reconstruction reach low PER in more seeds than wav2vec-U 2.0? Collapse rate across seeds is the stability metric.
- Boundary alignment: do the learned $\alpha$ peaks align with TIMIT boundaries better than stride 3 and comparably to REBORN?
- Scaling: do memory and throughput stay linear up to minute-long inputs, in contrast to soft-DTW?
If the first two checks fail while the stride-plus-reconstruction ablation succeeds, we conclude that differentiable segmentation is not what unsupervised ASR lacks.
References
- Towards End-to-End Unsupervised Speech Recognition
- REBORN: Reinforcement-Learned Boundary Segmentation with Iterative Training for Unsupervised ASR
- End-to-end Adversarial Text-to-Speech
- A Differentiable Alignment Framework for Sequence-to-Sequence Modeling via Optimal Transport
- Soft-DTW: a Differentiable Loss Function for Time-Series
- Learnable Dynamic Temporal Pooling for Time Series Classification
- Differentiable Segmentation of Sequences
- Unsupervised Speech Recognition