Speech systems that skip text increasingly turn audio into a short string of discrete "units," roughly like learned phonemes, and then model those strings the way text models model words. A recent method, duration-penalized dynamic programming (DPDP), makes these strings about 40% shorter. Dynamic programming here means picking the best whole sequence of labels at once rather than labelling each frame separately. DPDP charges a fixed fee every time the label changes between neighbouring 20 ms frames, so brief flickers get absorbed into longer units. That fee was chosen on clean, read, standard English. Many regional and rural varieties differ from the standard exactly in short, fast events: glides, clipped or lengthened vowels, rapid elisions in spontaneous talk. A single fee tuned on standard speech may therefore merge away the sounds that distinguish a dialect, and so build geographic bias into the tokenizer before any model is trained.
This note proposes two things. The first is a measurement: does DPDP's compression cost underrepresented regions more than well-represented ones, at matched bitrate? The second is a fix. The fee to switch units should depend on how fast the signal is actually changing at that moment, measured from the audio itself, and should not depend on a dialect label. If this works, it gives a compression scheme that keeps the 40% shortening while keeping per-region errors close to those of uncompressed units. It also needs no geographic metadata, which matters because dialects form continuums rather than categories.
Core Claim
Spoken Language Modeling with Duration-Penalized Self-Supervised Units supplies the method. It quantizes self-supervised speech features (WavLM features clustered with K-means) by minimizing distance-to-codeword plus a scalar penalty $\lambda$ per code switch. It reports shorter sequences and better per-bit language modeling, but also no advantage or a degradation on phone discrimination. That last result is the motivating evidence: coarser units already blur short phonetic events on LibriSpeech. The paper evaluates only LibriSpeech.
Our hypothesis is that this blurring falls unevenly across regions. We also hypothesize that a locally adaptive penalty removes most of the uneven part. The adaptive objective over frames $t=1..T$ with features $h_t$, codewords $c_k$ and chosen codes $k_t$ is
$$\min_{k_{1:T}} \sum_{t=1}^{T} \lVert h_t - c_{k_t}\rVert^2 + \lambda_0\, e^{-\beta s_t}\,\mathbb{1}[k_t \neq k_{t-1}],$$
where $s_t$ is the standardized local rate of feature change. Concretely, $s_t$ is the cosine distance between $h_{t-1}$ and $h_{t+1}$, z-scored within the utterance. $\lambda_0$ and $\beta$ are two global scalars tuned once on standard speech. When the signal is moving fast, switching becomes cheap. In steady stretches, switching stays expensive. Within-utterance normalization makes the fee follow each speaker's own tempo. It still decodes exactly with the same Viterbi recursion, so the cost is unchanged. Setting $\beta=0$ recovers DPDP.
The pitch-tracking literature motivates the idea that a continuity prior must bend for fast dynamics. A Time-Warping Pitch Tracking Algorithm Considering Fast f0 Changes shows that stationary-frame Viterbi tracking breaks on fast f0 movement and fixes it with explicit time warping. We take the cheaper route of modulating the transition cost instead of warping time.
The proposal differs from the closest dialect work in a specific way. Accented Speech Recognition With Accent-specific Codebooks adds per-accent codebooks inside a supervised Conformer and decodes over five seen accents. It needs accent labels and transcripts, and it treats accents as buckets. Our method works at the unsupervised tokenizer, uses no labels, and is evaluated on regional continuums.
Research Program
The first stage is a bias audit. Using one multilingual self-supervised backbone (XLS-R features), we build K-means codebooks of size $K \in \{100, 500\}$ and quantize three ways at matched bitrates: plain frame-wise K-means with run-length deduplication, DPDP with a global $\lambda$, and the adaptive variant. We do this on three geographically structured corpora.
STT4SG-350 covers all seven Swiss German dialect regions with a parallel test set of 3,515 sentences per region. The parallel design is ideal because content is held fixed and only region varies. IndicVoices has 145 districts and is 74% extempore speech. It tests the claim that spontaneous tempo variation inside a region matters, not only variation between regions. VIVALDI, from Speech Analysis of Language Varieties in Italy, covers 17 Italian varieties under a leave-city-out protocol. That paper also constrains our evaluation. Its finding that models memorize city acoustics rather than regional phonetics means every regional comparison must hold out cities or speakers, or we risk measuring channel artifacts.
The second stage is the fix and its analysis. We tune $\lambda_0$ and $\beta$ on LibriSpeech, as the original DPDP was tuned, and transfer them unchanged. We then examine which segments the methods treat differently: segment-duration distributions per region, and which unit boundaries DPDP removes that the adaptive version keeps.
Three measurements are used throughout. The first is unit-based recognition: a small encoder trained on unit sequences to output transcripts, with word error rate (WER) reported per region or district. The second is a variety probe, a light classifier on unit histograms and bigrams under leave-city-out on VIVALDI. It asks how much regional information survives quantization. The third is the per-region disparity, defined as worst-region minus best-region WER and the gap's change relative to uncompressed units.
Baselines
The naive reference point is frame-wise K-means with consecutive duplicates collapsed. This is the standard unit pipeline, it has no transition cost, and it fixes the "nothing removed except exact repeats" floor. Every compressed system is compared to it at matched bitrate, and the cost of compression is read off as the difference.
The strongest existing compression method is global-$\lambda$ DPDP from Spoken Language Modeling with Duration-Penalized Self-Supervised Units, re-run on identical features and codebooks with $\lambda$ swept to hit the same bitrates as our method. That paper reports 40% length reduction at $K=500$ and 190 bits/sec on LibriSpeech. The file reports no numeric ABX, WER or language-modeling values, so no reported result was found on Lacuna for DPDP accuracy on any dialect corpus. Beating it means equal or lower WER on the worst-represented regions at the same bitrate, with standard-speech performance not worse.
A strong oracle competitor is DPDP with $\lambda$ grid-searched separately per region using that region's held-out data. It has the same total tuning budget per region as our two global scalars receive on LibriSpeech, plus the advantage of in-domain labels. Matching it without region labels counts as success. Beating it on IndicVoices, where within-district tempo varies, would support the claim that local adaptation captures more than a per-region constant.
The isolating ablation keeps $\lambda_t$'s overall distribution but shuffles $s_t$ across frames within each utterance. If gains survive shuffling, they come from penalty variance rather than from tracking the signal's speed.
As continuous-feature upper references, not competitors, we use two reported numbers. The fine-tuned XLS-R 1B reaches WER 14.0% and BLEU 74.7 on STT4SG-350 (STT4SG-350). XLSR-53-ITA with Multi-Similarity loss reaches macro F1 51.29% and accuracy 60.49% on VIVALDI leave-city-out (Speech Analysis of Language Varieties in Italy). Discrete units will likely fall below these. The question is how much each quantizer gives up, and for whom.
Why not just…?
Why not just tune $\lambda$ per region? This is the real threat, and it is why the per-region oracle is a baseline rather than a strawman. It needs region labels, and as DADA and the contiguous-region confusions in the VIVALDI study both stress, regions are a continuum and labels for rural speakers are often unavailable. If the oracle matches our method everywhere, including on spontaneous IndicVoices, we will report that the bias is a per-region constant. That is still a useful finding, and it would demote the adaptive penalty to a label-free approximation.
Why not just skip duration penalties and use plain K-means? DPDP's own results show plain units are longer and worse per bit for language modeling, and its 40% shortening is what makes spoken language models trainable. Our audit measures directly whether plain units are actually fairer across regions at matched bitrate. If they are, and the adaptive version cannot close the gap, the honest conclusion is that compression and regional fidelity trade off.
Why not just train the codebook on dialect data? That changes the codewords, while the penalty problem concerns the switching fee. The two are orthogonal. We therefore run every comparison twice, with codebooks fit on LibriSpeech and with codebooks fit on pooled in-domain training audio. If in-domain codebooks alone erase the regional gap for global DPDP, the adaptive penalty is unnecessary, and we will say so.
Why not just use a bigger model? The file's evidence argues against scale alone. On spontaneous IndicVoices speech, Whisper, USM and MMS WERs "often exceeded 40%", while a 130M Conformer trained on regional data reached 12–25% (IndicVoices). The Italian-adapted XLSR-53 beat the larger XLSR-128 on VIVALDI. Scale also does not touch the tokenizer, which is where the loss we target occurs.
Proposed Experiments
The decisive check uses the STT4SG-350 parallel test set. If global DPDP's WER increase over plain units, at matched bitrate, is significantly larger for some dialect regions than others, the bias exists. Significance is assessed by a speaker-level bootstrap, since sentences are fixed across regions. If the adaptive penalty shrinks that spread without raising LibriSpeech-side error, the fix works.
The second check runs on IndicVoices extempore audio, grouped by district. It compares adaptive DPDP against per-district-tuned DPDP to separate within-region tempo effects from between-region effects.
The third check is the leave-city-out variety probe on VIVALDI. Retaining more regional information after compression is a gain only if it does not come from city-specific channel cues. We test this by checking whether the probe still works when trained and tested on disjoint cities.
The shuffled-$s_t$ ablation is the final check. It must lose the adaptive variant's advantage for the mechanism claim to stand. A null result across all three corpora, where global DPDP turns out to be region-neutral, would also be publishable. It would clear a widely used compression step of a plausible fairness concern before it spreads to multilingual spoken language models.
References
- Spoken Language Modeling with Duration-Penalized Self-Supervised Units
- A Time-Warping Pitch Tracking Algorithm Considering Fast f0 Changes
- Accented Speech Recognition With Accent-specific Codebooks
- STT4SG-350: A Speech Corpus for All Swiss German Dialect Regions
- IndicVoices: Towards building an Inclusive Multilingual Speech Dataset for Indian Languages
- Speech Analysis of Language Varieties in Italy
- DADA: Dialect Adaptation via Dynamic Aggregation of Linguistic Rules