Speech is long and text is short. A second of audio yields about 100 feature frames, but only a handful of words. Speech translation systems pay for this gap in memory and in data efficiency, because the translation model has to learn on its own which frames belong together. One fix works well when source transcripts exist. An intermediate layer of the speech encoder is trained with connectionist temporal classification (CTC), a loss that lets a network emit a short label sequence from a long input without being told where each label starts. The encoder then merges consecutive frames that receive the same predicted label, so the rest of the model sees roughly one vector per sound. For low-resource and unwritten languages, however, the phone labels this fix needs are exactly what is missing.
We propose to replace those phone labels with self-supervised discrete speech units. These are cluster IDs obtained by running k-means over the internal features of a model such as HuBERT or XLS-R, which were pre-trained on unlabeled audio. The unit-based CTC head acts as a learned, transcript-free segmenter inside a compact speech translation model. The large self-supervised model is used only to produce training targets and is discarded at inference. The claim is specific and testable. At the same amount of paired data and the same sequence compression, segments induced by unit-CTC should give better translation than fixed downsampling, arbitrary coarse CTC labels, or naive merging of per-frame unit predictions, and the advantage should grow as paired data shrinks.
Core Claim and Relation to Prior Work
Two results make the idea plausible, and they pull in different directions. CTC-based Compression for Direct Speech Translation puts a phone-level CTC loss at encoder layer 8 and average-pools runs of identical predictions. This gains 1.3–1.5 BLEU on MuST-C and saves 10–16% memory, but the phone labels come from the Gentle aligner and a pronunciation dictionary. That paper supplies our mechanism, and its dependency on phone labels is the gap we target. Efficient CTC Regularization via Coarse Labels for End-to-End Speech Translation shows that CTC regularization works even with arbitrary labels: vocabulary IDs taken modulo 256 reach 24.3 BLEU on MuST-C En-De, versus 23.8 for full-vocabulary CTC. This result constrains us. If arbitrary labels regularize as well as meaningful ones, any gain we see from units must come from segmentation, not regularization, and the experiments must separate the two.
The method for obtaining targets comes from the discrete-unit literature. Compact Speech Translation Models via Discrete Speech Units Pretraining trains a small filterbank encoder to predict HuBERT k-means units (k=1000) with an autoregressive cross-entropy decoder, then fuses it into a compact translation model that needs no self-supervised model at inference. We keep their compact, teacher-free-at-inference setting. We change the pre-training objective from sequence generation to CTC, and we use the resulting alignments to shorten the sequence, which their model never does. Improving Textless Spoken Language Understanding with Discrete Units as Intermediate Target likewise uses units as an auxiliary target through an autoregressive decoder, and only for intent and slot prediction. DUB and BT4ST use units for back-translation and simply deduplicate consecutive IDs. These works show that units carry the linguistic content and discard speaker and noise, which motivates using them. None of them learns an alignment. Pre-training for Speech Translation: CTC Meets Optimal Transport argues that CTC pre-training teaches the encoder alignment itself, which is our reason to prefer CTC over cross-entropy, but it depends on source transcripts. We deliberately stay with an autoregressive translation decoder. We do not adopt the non-autoregressive route of CTC-NAST or CTC-S2UT, because the latter needs distillation from an autoregressive teacher that is hard to train in low-resource regimes. Mixing both changes would make the result uninterpretable.
Research Program
The model is a compact filterbank encoder-decoder of the size used by the DSU-Adapter. Units come from k-means over a middle layer of HuBERT-Base for English source audio and of XLS-R for other source languages; XLS-R was pre-trained on 128 languages, so it plausibly transfers to unseen ones. Clustering is fit on untranscribed source-language audio only.
At an intermediate encoder layer $\ell$, a CTC head predicts the deduplicated unit sequence over $k$ units plus a blank symbol. A collapse block then drops frames whose blank probability exceeds a threshold, following Blank Collapse, and average-pools runs of identical non-blank predictions. The upper encoder layers and the decoder therefore operate on roughly one vector per acoustic unit.
Training has two stages. First, the lower encoder is pre-trained with unit-CTC on all available untranscribed source audio, which in the low-resource regime is far more plentiful than translated audio. Second, the full model is fine-tuned on paired audio and translation, with the unit-CTC loss kept as an auxiliary term.
We vary three things, because the file flags each as open for units: the codebook size $k$ (100, 500, 1000), the layer $\ell$ (the compression paper found layer 2 too early and needed layer 8), and the self-supervised layer used for clustering.
Beyond BLEU, we analyze segment quality directly. On English, where forced phone alignments are obtainable, we measure how well the unit-CTC segment boundaries match phone boundaries. For every method we also measure how often a single segment spans more than one phone, alongside the achieved compression ratio.
Baselines
The naive reference point is the same compact encoder-decoder trained from scratch on paired data with fixed convolutional downsampling and no auxiliary loss. On CoVoST-2 X→En, the DSU-Adapter paper reports 8.14 average BLEU for this from-scratch setting and 9.00 for ASR pre-training.
The strongest existing transcript-free compact method is the DSU-Adapter itself, which reports 9.13 average BLEU over 21 CoVoST-2 X→En pairs. It is the right competitor because it uses the same teacher units and the same no-teacher-at-inference constraint, and differs only in using cross-entropy with no compression. We rerun it with identical units, encoder size, pre-training audio, fine-tuning data and tuning budget. Beating it means a higher average BLEU with fewer encoder-output frames, with the gain concentrated on the lowest-resource CoVoST-2 languages.
Phone-supervised CTC compression, following the compression paper (21.9 BLEU on MuST-C En-De), serves as an upper reference on English, where transcripts exist. Matching it without transcripts would be the headline result.
CoLaCTC with 256 modulo labels (24.3 BLEU on MuST-C En-De) is run as an auxiliary loss on the same compact model. It needs text labels but no pronunciation dictionary, so it isolates regularization from acoustic segmentation.
Two ablations isolate our key ingredient, and both match our method's compression ratio exactly. In the first, units are remapped by modulo into arbitrary label sets: this keeps the segment boundaries but destroys the acoustic meaning of the labels. In the second, the segment boundaries from unit-CTC are kept but every segment gets a random label, so that boundaries and label identity can be credited separately.
As a non-compact ceiling, we report fine-tuned XLS-R numbers where we can afford them, labeled as such.
Why not just…?
Why not just deduplicate per-frame unit predictions without CTC? k-means units are already aligned to frames at 50 Hz, so one could train frame-level cross-entropy against unit IDs and merge equal adjacent predictions. This is the most serious threat. Our argument against it is that k-means assignments flicker between neighboring clusters at transitions, which should fragment segments. CTC's blank symbol can absorb those transition frames. We test this directly with a frame-level cross-entropy-plus-merge baseline at matched compression. If it matches unit-CTC on BLEU across all data sizes, we will report that CTC adds nothing here and reframe the contribution as learned unit-based compression in general.
Why not just fine-tune a large multilingual model like XLS-R? It is stronger when memory is no object; XLS-R 2B reports +7.4 BLEU average on CoVoST-2 X→En. Our setting is compact deployment, where the DSU-Adapter paper argues these models are infeasible. In addition, DUB finds that XLS-R plus mBART loses to unit-based training below 2 hours of paired data. We report the ceiling rather than claim to beat it.
Why not just use fixed downsampling or arbitrary CoLaCTC labels? CoLaCTC shows that arbitrary labels regularize well, so a gain from units could be pure regularization. The modulo and random-label ablations, together with a fixed-stride baseline at identical sequence length, answer this. If they tie with unit-CTC, acoustic grounding does not matter.
Why not just use a universal phone recognizer? AlloST raises Fisher 40-hour BLEU from 21.06 to 29.49 with Allosaurus phones. However, that recognizer was trained with supervision on phonetic data from over 2,000 languages. We include it on Fisher as a supervised-prior reference and test whether self-supervised units close the gap.
Proposed Experiments
We evaluate on four settings. The first is CoVoST-2 X→En, all 21 languages, split into high- and low-resource groups following CoVoST 2. The second is MuST-C En-De and En-Fr with paired data subsampled to 10, 50, 100 hours and full; BT4ST reports its largest gain (+5.4 BLEU) at 100 hours. The third is the Fisher Spanish-English 40-hour subset. The fourth is Mboshi-French (4 hours), where CoVoST-2 fine-tuning reports 12.6 BLEU. We report BLEU and chrF, the encoder output length after compression, peak training memory, and decoding speed.
The decisive checks are three. First, the BLEU-versus-paired-hours curve: unit-CTC compression should widen its lead over the DSU-Adapter and over fixed downsampling as hours decrease. If the curves are parallel or cross, the claim fails. Second, at a matched compression ratio, unit-CTC should beat both frame-CE merging and the random-label and modulo ablations. Third, on MuST-C En-De, the gap to phone-supervised CTC compression tells us how much of the phone dictionary's value self-supervised units recover.
For Mboshi and Taigi-type settings, no reported result was found on Lacuna for a compact transcript-free model, so our from-scratch and DSU-Adapter reruns define the reference.
References
- CTC-based Compression for Direct Speech Translation
- Efficient CTC Regularization via Coarse Labels for End-to-End Speech Translation
- Compact Speech Translation Models via Discrete Speech Units Pretraining
- Improving Textless Spoken Language Understanding with Discrete Units as Intermediate Target
- DUB: Discrete Unit Back-translation for Speech Translation
- Back Translation for Speech-to-text Translation Without Transcripts
- Pre-training for Speech Translation: CTC Meets Optimal Transport
- CTC-based Non-autoregressive Speech Translation
- CTC-based Non-autoregressive Textless Speech-to-Speech Translation
- XLS-R: Self-supervised Cross-lingual Speech Representation Learning at Scale
- Blank Collapse: Compressing CTC Emission for the Faster Decoding
- AlloST: Low-Resource Speech Translation Without Source Transcription
- CoVoST 2 and Massively Multilingual Speech Translation