Speech recognizers do not write out words letter by letter. They emit chunks of spelling called subwords, and those chunks are normally chosen by counting which letter sequences are frequent in text. Frequency in text says nothing about sound. A chunk can cut through the middle of one sound, or glue together letters that are pronounced differently in different words, and the acoustic model then has to learn a mapping from audio to units that the audio does not support. Two papers show that choosing subwords by pronunciation fixes much of this. Improving End-to-end Speech Recognition with Pronunciation-assisted Sub-word Modeling reports 21.3% word error rate (WER) on LibriSpeech 100h test-clean against 29.5% for text-frequency subwords. Phonetically Induced Subwords for End-to-End Speech Recognition reports 9.07% against 10.45% on the full 960h set. Both methods need a pronunciation dictionary, and the languages that would gain most are the ones that lack one.
We propose to induce pronunciation-grounded subwords from the target language's own small set of transcribed recordings, with no dictionary. The language's audio, aligned to its spelling through a multilingual phonetic encoder, stands in for the lexicon. A letter sequence becomes a unit only if it sounds the same each time it occurs. Units may also contain the word-boundary symbol when speakers acoustically fuse adjacent words. The recognizer is trained by summing over every way of splitting a transcript into these units, so no single tokenization is imposed. In short, the audio decides both what the units are and where words begin and end. We test whether this beats character-level and frequency-subword recognizers on spontaneous, dialectal Hindi telephone speech and on low-resource FLEURS languages.
Core Claim
In low-resource ASR, the output vocabulary can be made phonetically consistent without a lexicon. Pronunciation-assisted subword modeling decides which letter groups to merge by checking how consistently they align to phonemes in a dictionary. We run the same consistency test on alignments between transcripts and frame-level universal phone predictions. Doing so should recover most of the gain that dictionary-based phonetic subwords give over frequency subwords. A second claim follows. If the space character is allowed inside units, and units are kept only when they are acoustically consistent, the decoder places word boundaries itself, and those placements reflect evidence from the audio. This addresses a problem that Open vocabulary speech recognition with flat hybrid models explicitly left open. That paper decoded words and letter-phoneme pairs together in one search space and cut errors per out-of-vocabulary word from 1.7–2.0 to 1.2–1.5. However, when several sub-word pieces were decoded in a row, it could only guess where the word breaks fell, and it merged them by a concatenation heuristic.
Where This Sits in Prior Work
The proposal borrows its unit-selection principle from the two phonetic-subword papers above. Both select units by letter-to-phoneme consistency using fast_align over a dictionary. We keep the principle and replace the dictionary with audio. Phonetic-assisted Multi-Target Units Modeling for Improving Conformer-Transducer ASR system supplies an architectural lesson. It supervises lower encoder layers with phonetic units through an auxiliary loss and upper layers with text-frequency subwords, reaching 9.6% on LibriSpeech 100h test-clean. It is English-only, depends on a dictionary, and leaves word boundaries to whitespace in the subwords.
Two papers on multilingual phonetic encoders provide our acoustic backbone and show why our design differs.
- Language-universal Phonetic Encoder for Low-resource Speech Recognition pretrains an encoder on audio-to-IPA targets, IPA being the International Phonetic Alphabet. During target fine-tuning it then discards the phonetic decoder.
- Language-Universal Phonetic Representation in Multilingual Speech Pretraining for Low-Resource Speech Recognition uses an IPA labeler to pseudo-label untranscribed audio for HuBERT pretraining. It reports matching standard HuBERT with 75% less supervised data.
Both throw the phonetic signal away exactly where the target language's spelling is decided. We keep it, both as an auxiliary loss and as the source of alignments for unit induction.
The training objective comes from Training Hybrid Language Models by Marginalizing over Segmentations. That work sums, by dynamic programming, over all character and n-gram decompositions of text, but only for text language modeling. We apply the same summation inside an acoustic sequence loss.
Two strands of work constrain the design.
- Unsupervised Word Segmentation from Discrete Speech Units in Low-Resource Settings shows that units discovered purely from speech by neural quantizers yield sequences that are too long and boundaries that are poor. On Mboshi it reports 52.7% boundary F-score for VQ-VAE and 64.7% for the best Bayesian model. We therefore do not discover units from audio alone; we anchor them to whatever spelling exists.
- Deriving Phonetic Transcriptions and Discovering Word Segmentations for Speech-to-Speech Translation in Low-Resource Settings reports collapse to 6.97 BLEU when an English phone recognizer is applied to Mandarin. A single-language phone recognizer is therefore a poor alignment source, which is why we use a multilingual IPA encoder adapted to the target language.
When Tokenizers Fail: Byte-Level Chunking for Zero-Shot Transfer to Low-Resource Languages motivates the problem from the text side. It documents a Qwen3 tokenizer splitting 97.7% of Bhojpuri words, and it learns word-aligned byte chunks, but from text alone. Our chunks are learned from sound.
Research Program
The system has three stages.
Stage 1: alignment model. We fine-tune a multilingual IPA-pretrained encoder, built in the manner of the language-universal phonetic encoder, on the target language's transcribed audio. It has two output layers trained with CTC (connectionist temporal classification, a loss that lets a model emit a label sequence without frame-level alignments):
- a character head on the top layer;
- a universal-phone head on a lower layer, with targets from the multilingual IPA labeler's pseudo-labels, so no target-language lexicon is used.
Forced alignment with both heads gives, for every training utterance, character spans and the phone strings realized over the same frames.
Stage 2: unit induction. For each candidate letter string $g$ we collect the phone strings aligned to its occurrences. Candidates are all character n-grams up to length six, including ones that contain the space symbol. We score each candidate by the normalized entropy
$$H(g) = -\frac{1}{\log K_g}\sum_{p} q_g(p)\log q_g(p),$$
where $q_g(p)$ is the fraction of occurrences of $g$ realized as phone string $p$, and $K_g$ is the number of distinct phone strings observed for $g$. Units are added greedily by frequency among candidates with $H(g)$ below a threshold $\tau$ and count above a minimum, up to a vocabulary budget. All single characters are always included, so the vocabulary stays open. A candidate spanning a space survives only if the fused pronunciation is itself consistent, which is what we expect at sandhi-like junctions where adjacent words merge in speech.
Stage 3: recognizer training. The final recognizer keeps the phone head as an auxiliary loss. Its top head is trained with
$$\mathcal{L}(x,y) = -\log \sum_{s \in S_V(y)} P_{\text{CTC}}(s \mid x),$$
where $x$ is the audio, $y$ the character transcript, $S_V(y)$ the set of all ways to write $y$ as a sequence of units from vocabulary $V$, and $P_{\text{CTC}}$ the standard CTC probability of unit sequence $s$. This can be computed by composing the CTC topology with the unit-to-character transducer.
Decoding. Words come from the space symbols the model emits. An optional word-level language model with a unit-level fallback, in the spirit of the flat hybrid recognizer, rescores hypotheses, but it no longer has to guess boundaries.
The main deliverables are three things. The first is the recognizer. The second is a measurement of how closely audio-induced units match dictionary-induced ones where a dictionary exists. The third is an analysis of where cross-word units appear and whether they help.
Baselines
The naive reference point is a character-level CTC head on the same multilingual phonetic encoder. It uses the same target data, the same encoder initialization, the same number of training steps, and the same language model at decoding. This baseline is the real threat. It needs no unit design at all, and on LibriSpeech 100h characters already beat frequency subwords (23.2% against 29.5% WER in the pronunciation-assisted subword paper). Beating it is the minimum bar for any claim.
The second baseline uses SentencePiece frequency subwords at a vocabulary size tuned on the development set, again on the same encoder. We run it both with deterministic tokenization and with the same segmentation-summing loss we use, so that the summation and the phonetic selection are credited separately.
The third baseline is the multi-target design from the phonetic-assisted multi-target paper, adapted to the target language. The lower layer is supervised with universal phones and the upper layer with frequency subwords. This is the strongest existing method that keeps phonetic supervision. Its reported result is 9.6% WER on LibriSpeech 100h test-clean, but only in English with a dictionary.
On the primary benchmark, Gram Vaani Hindi spontaneous telephone speech, the strongest reported systems come from Gram Vaani ASR Challenge on spontaneous telephone speech recordings in regional variations of Hindi, all trained on 100h of labeled speech:
| System | Eval WER | Eval CER |
|---|---|---|
| Kaldi hybrid TDNN-F | 29.7% | 15.1% |
| ESPnet Conformer | 32.9% | 19.0% |
| wav2vec 2.0 pretrained on 1,000h of in-domain unlabeled audio | 35.8% | not reported in the file |
We quote these numbers as external references. Our claims rest on the controlled comparisons above, where every system shares data, encoder, step budget, and an equal hyperparameter search of about twenty trials each.
Beating the controlled baselines means a WER reduction on the evaluation set that a paired bootstrap test finds significant at p < 0.05. Against the character baseline, we treat a relative reduction of at least 5% as meaningful. Ablations remove one ingredient each:
- phone-consistency selection, replaced by frequency at an equal vocabulary size;
- the segmentation summation, replaced by a single greedy segmentation;
- cross-word units;
- the retained phone head.
Why not just…?
Why not just fine-tune a large multilingual self-supervised model on characters? This is the strongest alternative, and our character baseline on the same encoder is exactly this system. The file's evidence suggests raw self-supervised pretraining is not enough on spontaneous dialectal speech: Gram Vaani's wav2vec 2.0 at 35.8% WER trails the TDNN-F system at 29.7%. That comparison involves different encoders, though, so it does not settle the question. If our units fail to beat characters on the same encoder, the unit-induction claim fails and we will report it as such.
Why not just build a pseudo-dictionary with a universal grapheme-to-phoneme tool and run the existing phonetic-subword method? Where such a tool covers the language, this is a fair competitor. We will run it on the FLEURS languages where one is available. The same dictionary is the bottleneck the multilingual encoder papers themselves depend on. It reflects the standard spelling-to-sound mapping rather than the regional pronunciation actually heard in Gram Vaani recordings, and it does not exist for most of the world's languages. If it matches our method where available, our contribution narrows to languages without such a tool, and we will say so.
Why not just sum over frequency-subword segmentations? That alternative tests whether the gain comes from removing the fixed tokenization rather than from phonetics. The frequency-units-with-summation ablation answers it directly, and it is why that ablation sits next to our method in every table.
Why not just discover units from audio with a neural quantizer? The unsupervised word segmentation paper shows neural quantizers produce overly long sequences and weak boundaries, and they yield no spelling. Our units are always character strings, so the recognizer still writes text.
Proposed Experiments
Experiment 1: lexicon check on English. On LibriSpeech 100h, where a dictionary exists, we induce units from audio and compare them with dictionary-based pronunciation-assisted units at matched vocabulary size, measuring both inventory overlap and WER. The original paper reports 21.3% test-clean for its system, which we reimplement in our architecture for a fair comparison. If audio-derived units recover less than half of the gap between frequency and dictionary-based units, the core premise fails.
Experiment 2: main benchmark, Gram Vaani. We train on the 20h reliable subset and on the full 100h, using the 1,000h unlabeled portion only for encoder adaptation in all systems. We report WER and CER on the 3h evaluation set.
Experiment 3: low-resource FLEURS languages. We use FLEURS languages from the South Asian group with simulated budgets of 5, 10, and the full amount of transcribed audio, reporting CER. FLEURS: Few-shot Learning Evaluation of Universal Representations of Speech reports a best-baseline average CER of 14.1% across languages.
Experiment 4: data-budget curve. We plot the gain over characters against the amount of transcribed data. We expect the gain to shrink as data grows, by analogy with the roughly 4,000-example ceiling that Multilingual unsupervised sequence segmentation transfers to extremely low-resource languages found for cross-lingual segmentation transfer. A flat curve would instead suggest the gain does not come from compensating for scarcity.
Experiment 5: boundary analysis. We score the decoder's emitted spaces against reference word boundaries and count boundary errors separately from substitutions. We also check whether utterances with cross-word units have lower boundary error than the same model with those units removed. For a decision rule, we compare against the best unit-free system across Experiments 2 and 3. If cross-word units do not improve boundary accuracy in Gram Vaani or on any FLEURS language, the second claim is dropped and the method is presented purely as lexicon-free phonetic subword induction.
References
- Improving End-to-end Speech Recognition with Pronunciation-assisted Sub-word Modeling
- Phonetically Induced Subwords for End-to-End Speech Recognition
- Open vocabulary speech recognition with flat hybrid models
- Phonetic-assisted Multi-Target Units Modeling for Improving Conformer-Transducer ASR system
- Language-universal Phonetic Encoder for Low-resource Speech Recognition
- Language-Universal Phonetic Representation in Multilingual Speech Pretraining for Low-Resource Speech Recognition
- Training Hybrid Language Models by Marginalizing over Segmentations
- Unsupervised Word Segmentation from Discrete Speech Units in Low-Resource Settings
- Deriving Phonetic Transcriptions and Discovering Word Segmentations for Speech-to-Speech Translation in Low-Resource Settings
- When Tokenizers Fail: Byte-Level Chunking for Zero-Shot Transfer to Low-Resource Languages
- Gram Vaani ASR Challenge on spontaneous telephone speech recordings in regional variations of Hindi
- FLEURS: FEW-Shot Learning Evaluation of Universal Representations of Speech
- Multilingual unsupervised sequence segmentation transfers to extremely low-resource languages