Measured Rhythm and Pitch Profiles as a Prior for Few-Shot Accented Speech Recognition

When a speech recognizer meets a new non-native speaker, it usually gets only a handful of that person's sentences to adapt on. Phonetics research has shown that much of a foreign accent is shared by speakers of the same native language. The mismatch lies in rhythm and pitch: Mandarin learners of English give every syllable about the same length and put a small pitch peak on nearly every syllable, while native speakers shorten unstressed vowels and spread pitch movement across the phrase. These habits can be summarized by a few interpretable numbers, such as how much vowel durations vary from one syllable to the next or how many pitch peaks occur per syllable. They can also be measured from a few seconds of audio. Current few-shot adaptation methods for recognition ignore this. They either fit a small module to the new speaker from scratch or condition on a learned "accent embedding" whose meaning is opaque and which has only seen the accents present in training.

We propose to compute a low-dimensional prosodic profile for each speaker, a vector of established rhythm and pitch metrics, and use it as the input to a small network that sets the weights of adapter layers inside a frozen recognizer. The profile supplies the part of the adaptation that speakers of a similar native-language rhythm type share. The speaker's few enrollment sentences are then spent only on what is truly individual. The bet is that this measured, typology-grounded conditioning generalizes to native languages never seen in training better than learned accent embeddings do, and that it degrades more gracefully when enrollment shrinks to two to eight utterances. Both claims can be tested on public corpora, and both could fail cleanly.

Core Claim

Rhythm and pitch transfer from a speaker's first language is systematic, measurable, and shared across speakers with similar native-language prosody. A recognizer whose adapters are predicted from a measured prosodic profile should therefore transfer to unseen first languages. Adapters conditioned on categorical accent labels or on learned accent vectors should transfer less well. This advantage should be largest exactly where per-speaker data is scarcest.

The phonetics literature motivates this claim by showing that the relevant quantities are both discriminative and compact. Manual segmentation of Mandarin child learners' English found that the normalized pairwise variability of vowel durations and the normalized standard deviation of vowel durations were the only rhythm metrics that cleanly separated native, high-proficiency and low-proficiency groups. Vowel insertion after final stops inflated the proportion of vocalic time (A Comparison of English Rhythm Produced by Native American Speakers and Mandarin ESL Primary School Learners). An analysis of pitch contours extracted with REAPER and filtered at a 0.5-semitone prominence threshold found about 1.08 pitch peaks per syllable for low-proficiency Mandarin learners versus about 0.74 for native speakers. Peak density and the spread of peak spacing were the strongest predictors of proficiency (F0 Patterns of L2 English Speech by Mandarin Chinese Learners). Korean learners reduce unstressed vowels in isolated words but not in connected sentences, and a duration deviation of about 20% is enough for native listeners to hear "staccato" speech (What makes a non-native accent?: a study of Korean English). Listeners can identify a speaker's linguistic background from speech with all segmental content filtered out (Maclagan et al.), so this information is carried by prosody alone. Finally, rhythmic variation has been tied to recognition error disparities (Exploring Sources of Racial Bias in Automatic Speech Recognition through the Lens of Rhythmic Variation). That is the reason to expect these metrics to matter for word error rate and not only for perceived accent.

Relation to Prior Work

The closest recognition work conditions adapters on accent information, but never on measured prosody. Layer-wise fast adaptation mixes a set of small bottleneck adapters ("bases") with coefficients predicted from a continuous accent embedding, reporting a 12% relative word error rate reduction on AESRC2020. Its bases spontaneously clustered typologically similar accents (Layer-Wise Fast Adaptation for End-to-End Multi-Accent Speech Recognition). That clustering is the strongest hint that a typological conditioning variable would work. We reuse the basis-mixing mechanism and replace the learned embedding with the measured profile. Two other systems inject accent information through different representations. One feeds sentence-level accent vectors from a network trained on accent labels into a time-delay acoustic model (Achieving Multi-Accent ASR via Unsupervised Acoustic Model Adaptation). The other injects frame-level accent features with a loss that penalizes frame-to-frame variation (Improving Accent Identification and Accented Speech Recognition Under a Framework of Self-Supervised Learning). Prosodic transfer is concentrated on particular syllables, which is in tension with that uniformity loss. We sidestep the tension by pooling at the syllable and vowel level instead of forcing frames to agree.

On the speaker-adaptation side, Personality-memory Gated Adaptation is the nearest design. It inserts parallel adapters into a frozen Conformer recognizer and gates them with a prosody embedding from a vector-quantized autoencoder, reaching a 21.62% relative error reduction on regional Mandarin with 10 minutes per speaker (Personality-memory Gated Adaptation: An Efficient Speaker Adaptation for Personalized End-to-End Automatic Speech Recognition). Its prosody embedding is unconstrained and mixes voice quality with rhythm. It was also evaluated on dialects of one language rather than on cross-language transfer, and with ten minutes of audio rather than seconds. AdaSpeech supplies our per-speaker residual: it tunes only the gain and bias of conditional layer normalization, about 5,000 parameters, and reaches a similarity score of 3.96 versus 3.34 for a speaker-embedding baseline on VCTK. It also reports degradation below about 10 sentences (AdaSpeech: Adaptive Text to Speech for Custom Voice), and that degradation is precisely the regime where a shared prior should help.

The only work touching all three directions is AM-SIMSAM (Multi-Scale Accent Modeling with Disentangling for Multi-Speaker Multi-Accent TTS Synthesis). It separates speaker from accent with adversarial training on L2-ARCTIC, but it is a synthesizer and depends on categorical accent labels that cannot extend to unseen accents. Cross-lingual synthesis adaptation from phonological features works with two utterances but explicitly leaves the source-language prosodic accent uncontrolled (Cross-Lingual Low Resource Speaker Adaptation Using Phonological Features). Our proposal differs from all of these in three ways. It targets recognition. It conditions on an explicit, interpretable profile instead of a learned vector or a label. And it is evaluated specifically on held-out native languages at two to thirty-two enrollment utterances.

Research Program

The system has three parts. The first is a prosodic profile extractor. For a speaker with enrollment utterances and transcripts, we use the forced-aligned phone boundaries that L2-ARCTIC already provides, or a first-pass alignment for other corpora. From these we compute five rhythm measures: the normalized pairwise variability of vowel durations, the normalized standard deviation of vowel and consonant durations, the proportion of vocalic time, and silent-pause share. We then extract pitch with REAPER, convert it to semitones relative to the speaker's baseline, and compute four pitch measures following the F0 study: peaks per syllable at the 0.5-semitone threshold, the standard deviation of peak spacing, pitch range, and mean pitch slope within vowels. Each feature is z-scored against native reference speakers from L1-ARCTIC. This yields a profile $p_s$ of about ten dimensions per speaker $s$.

The second part is a profile-conditioned adapter. Into a frozen multilingual recognizer we insert $J$ basis adapters $B_1,\dots,B_J$ per encoder block, as in layer-wise fast adaptation. A two-layer network $g$ maps the profile to mixing weights:

$$\Delta_s = \sum_{j=1}^{J} \alpha_j(p_s)\, B_j + \delta_s,$$

Here $\Delta_s$ is the total adapter contribution for speaker $s$, $\alpha_j(p_s)$ is the softmax output of $g$ for basis $j$, and $\delta_s$ is the per-speaker residual. The bases and $g$ are trained jointly across many non-native training speakers. During this training, $\delta_s$ is held at zero, so the bases can only explain what the profile can predict.

The third part is the residual $\delta_s$ itself. It is implemented as AdaSpeech-style layer-normalization gain and bias, a few thousand parameters, and is the only component fitted on a new speaker's enrollment data. It is tuned with early stopping on a held-out enrollment utterance. With zero enrollment transcripts, the profile can still be estimated from a first-pass alignment, which gives a zero-shot variant.

Two analyses accompany the system. The first concerns measurement reliability: by bootstrapping over utterances, we measure how quickly the profile and a learned accent embedding each stabilize as enrollment grows from 2 to 32 sentences. A ten-number summary of durations and peaks should stabilize faster than a high-dimensional embedding, and if it does not, the few-shot argument weakens. The second asks which component carries the gain, attributing improvements separately to the rhythm block and the pitch block. It also checks whether profile distance from native predicts per-speaker word error rate before and after adaptation.

Baselines

The naive reference point is the frozen pretrained recognizer with no adaptation. For comparability with the closest published setting, we use the 300M-parameter MMS backbone fine-tuned on a 5-hour, 15-speaker native English budget, as in Effects of Speaker Count, Duration, and Accent Diversity on Zero-Shot Accent Robustness in Low-Resource ASR. That paper reports 15.6% character error rate on unseen L2-ARCTIC accents, against 19.8% with one speaker. Our reproduction must land near 15.6% before any comparison is trusted.

The second baseline is pooled multi-speaker fine-tuning. The same backbone is fine-tuned on exactly the non-native training speakers our hypernetwork sees, with no conditioning. This tests whether simply exposing the model to many L2 speakers, which the speaker-count paper found more valuable than curated accent diversity, already captures everything the profile provides.

The third baseline is per-speaker adapter fine-tuning without any shared prior: parallel adapters or layer-normalization gain and bias fitted on the speaker's enrollment utterances alone, in the spirit of PGA and AdaSpeech. This is the standard low-resource speaker adaptation recipe.

The strongest existing competitor is the multi-basis adapter of layer-wise fast adaptation, with mixing coefficients predicted from a learned accent embedding computed on the same enrollment audio. It is the strongest competitor because it shares our adapter architecture and differs only in the conditioning variable. Any gain we show over it is attributable to the profile. The paper reports a 12% relative word error rate reduction on AESRC2020. We found no reported result on Lacuna for this method on L2-ARCTIC or under few-shot per-speaker enrollment, so we will rerun it ourselves. We also run a PGA-style variant whose gate uses a learned autoencoder prosody embedding. That paper's 21.62% relative error reduction comes from a different language and ten minutes of audio, so it is context rather than a target. We found no reported result on Lacuna for few-shot (2–32 utterance) speaker adaptation of recognition on L2-ARCTIC.

Fairness is enforced by giving every system the same backbone, training speakers, enrollment utterances, adapter parameter count, and hyperparameter search budget: an equal number of trials on a validation split of training speakers only. The learned accent embedding is projected to the same dimensionality as the profile, so that size is not a confound.

Three ablations isolate the key ingredient. In the first, each speaker's profile is replaced by the profile of a random speaker with a different native language. If gains survive, the profile is acting as a regularizer, not as information. In the second, the profile is replaced by an oracle one-hot native-language label. This should match the profile on seen languages and fail on unseen ones; if it matches on unseen ones too, something is leaking. The third runs rhythm-only and pitch-only profiles separately.

Beating the strongest baseline means a lower word error rate on held-out native languages than the accent-embedding basis adapter. The difference must hold under a paired bootstrap over test utterances with speaker-level resampling. It must be present at 2 and 8 enrollment utterances, and it must cost no more than a 0.3-point absolute increase in source-domain word error rate on LibriSpeech, a tolerance chosen in the spirit of the 0.27% degradation PGA reports.

Why not just…?

Why not just fine-tune small adapters on each speaker's few sentences? This is the default, and at 32 utterances it may well be competitive. AdaSpeech itself reports that quality falls off below about 10 sentences, and an adapter fitted to two sentences has no basis for separating rhythm habits from chance. The enrollment-size sweep settles this directly. If per-speaker adapters match the profile-conditioned model at 2 and 8 utterances, the shared prior is not earning its keep, and we will say so.

Why not just condition on a learned accent embedding, as layer-wise fast adaptation and the unsupervised accent-embedding work do? This is the real threat. A self-supervised encoder may already encode rhythm and pitch implicitly, and the accent-embedding work showed that targeted accent embeddings beat generic speaker vectors. Our argument rests on two testable claims. The first is that a measured, typological profile extrapolates to unseen native languages, while a learned embedding mostly interpolates among training accents. The second is that the profile stabilizes from fewer utterances. The leave-one-native-language-out evaluation and the reliability bootstrap test exactly these claims. If the learned embedding matches the profile on unseen languages, the proposal reduces to an interpretability contribution, and we will report it that way rather than claim accuracy gains.

Why not just train on more non-native speakers? The speaker-count study found that more speakers beat more hours per speaker, and that curated accent diversity added little once speaker count was high. It is therefore plausible that a pooled model already covers the space of prosodic transfer. The pooled fine-tuning baseline uses exactly our training speakers. If it closes the gap on unseen languages, the right conclusion is that speaker count covers prosodic variation and explicit conditioning is unnecessary, which would itself answer an open question raised by that paper.

Why not just correct the transcripts with a large language model afterwards? In CantoASR, text-only correction was far worse than prosody-aware acoustic modeling: 27.3% character error rate for GPT-4o and 55.8% for LLaMA-3, against 11.19% for the prosody-aware system (CantoASR: Prosody-Aware ASR-LALM Collaboration for Low-Resource Cantonese). That was in a tonal language, so it does not transfer directly. Text correction is also orthogonal to our method and could be stacked on any of the systems above. We therefore treat it as out of scope rather than as a competitor.

Proposed Experiments

Training speakers for the bases and the profile-to-weights network come from AESRC2020's eight training accents (Layer-Wise Fast Adaptation for End-to-End Multi-Accent Speech Recognition), together with L2-ARCTIC speakers from all native languages except the held-out one (L2-ARCTIC: A Non-native English Speech Corpus). Evaluation is leave-one-native-language-out on L2-ARCTIC's six languages, each with four speakers. Each test speaker's 67 minutes are split into enrollment pools of 2, 8 and 32 utterances and a disjoint test set of held-out prompts. We also evaluate on AESRC2020's unseen Spanish and Canadian accents. Canadian speakers should receive a near-native profile and gain little from adaptation, which serves as a sanity check.

L2-ARCTIC has only four speakers per language, which confounds individual voice with native-language habits. We therefore add a large-scale unseen-accent test on AfriSpeech-200 (Advancing African-Accented English Speech Recognition: Epistemic Uncertainty-Driven Data Selection for Generalizable ASR Models). Its more than 120 accents, none present in training, are a much harder test of extrapolation from a typological profile. Metrics are word and character error rate, per-native-language and per-speaker breakdowns, source-domain error on LibriSpeech, and profile stability (variance across bootstrap resamples) as a function of enrollment size.

Three decisive checks determine the outcome. The proposal holds if the profile-conditioned model beats the accent-embedding basis adapter on held-out L2-ARCTIC languages and on AfriSpeech-200 at 2 and 8 enrollment utterances, if the shuffled-profile ablation erases the gain, and if the oracle-label ablation helps on seen but not unseen languages. It fails if the pooled baseline or the learned embedding matches it on unseen languages. A secondary check asks whether gains concentrate on speakers whose profiles lie far from the native reference, especially those with high peaks-per-syllable and low vowel-duration variability. That pattern is what the prosodic-transfer account predicts, and a generic regularization effect would not produce it.