Tone recognizers in languages like Mandarin and Vietnamese, where the pitch movement over a syllable changes word meaning, fail badly when moved to another dialect. A Vietnamese tone classifier that reaches about 78% accuracy within one dialect drops to 50–57% when trained on Northern speech and tested on Southern speech, or the reverse (Cross-Gender and Cross-Dialect Tone Recognition for Vietnamese). Most of the vocabulary is shared, and so are most of the tone categories. What changes is how each category is pronounced: a rise may start lower, a dip may lose its turning point, or two tones may sound the same. We propose to treat this shift as a small, structured remapping of each dialect's set of tone shapes. Fitting that remapping from unlabeled audio in the new dialect should recover much of the lost accuracy. It should also tell us which tones moved, how they moved, and which ones merged.
The idea is to place every syllable's pitch curve in a compact, speaker-normalized "contour space." Each tone of the source dialect becomes a cloud of points in that space. Unlabeled syllables from the target dialect form their own clouds. We then solve a matching problem that pairs source tones with target clouds under a constrained geometric transformation. Simply attaching pitch features to a classifier would not produce a readable map between the two dialects' tone systems, and this method does, so its output can be inspected and compared against phonetic descriptions.
Core Claim
Dialect shift in lexical tone is mostly a low-dimensional, per-tone warp of a shared contour inventory: changes in level, slope, and turning-point position, plus occasional many-to-one mergers. It is not an arbitrary change in the acoustic signal. If this holds, two consequences follow. First, an alignment restricted to that structure, estimated without target-dialect tone labels, should close most of the gap between cross-dialect and within-dialect accuracy. Second, it should beat both pitch normalization alone and generic unsupervised domain adaptation on neural features.
Phonetic work motivates this claim. Children exposed to the Changli dialect produce off-target Putonghua tones that are not random errors. They are blends, for example the pitch level and falling portion of Tone 3 combined with the slope of Tone 2 (Pitch Target Realization in Putonghua Tone Production of Children from Dialect-Speaking Regions). That is exactly the kind of partial, component-wise displacement a structured warp can represent. Wu tone sandhi patterns vary smoothly across geography once pitch is normalized against each speaker's long-term range (Dialect-geographical Acoustic-Tonetics: Five Disyllabic Tone Sandhi Patterns in Cognate Words from the Wu Dialects of ZhèJiāNg Province). This suggests that cross-dialect differences are well behaved in a properly normalized space.
These papers are descriptive and build no recognizer. The Vietnamese study documents the collapse but treats dialect as a test-time confound and does not try to adapt to it. Our proposal turns their observations into an adaptation method.
Research Program
Building the contour space. We first build the contour space from syllable-level pitch curves obtained by forced alignment. Each speaker's pitch is normalized with the extrinsic long-term z-score method of the Wu study rather than per-token normalization; that paper argues per-token normalization distorts cross-dialect comparisons when pitch registers differ. Curves are resampled to a fixed length and decomposed with functional principal component analysis (a method that summarizes curves by a mean shape plus a few orthogonal shape modes), following the children's-tone study. The basis is fitted on the source dialect. Each syllable becomes a short coefficient vector, and we append duration and harmonicity. Harmonicity matters because voice quality is a tone cue in its own right: creaky voice shifts Mandarin tone boundaries (The role of creaky voice in Mandarin tone 2 and tone 3 perception), and harmonicity added accuracy in Vietnamese.
Modeling the target dialect. Each source tone $t$ is summarized by a Gaussian with mean $\mu_t$. The unlabeled target syllables are fitted with a mixture whose components have means $\nu_k$. We then estimate a transport plan $\pi_{tk}$, meaning how much of source tone $t$ maps to target component $k$, together with transformation parameters $\theta$, by solving
$$\min_{\pi,\theta}\ \sum_{t,k} \pi_{tk}\, \lVert A_t\mu_t + b + c_t - \nu_k \rVert^2 \;+\; \lambda \sum_t \big(\lVert A_t - I\rVert^2 + \lVert c_t\rVert_1\big),$$
where $b$ is a shift shared by all tones (a global register or slope change), $A_t$ and $c_t$ are tone-specific deviations, and $\lambda$ controls how much individual tones may move. The L1 penalty encourages most tones to move only with the shared shift. We use unbalanced transport, which lets two source tones land on one target component, so a merger appears as a learned many-to-one correspondence rather than as classifier error.
Adaptation. We then map source-labeled data through the fitted transformation, or equivalently relabel target clusters, and retrain the tone classifier. The classifier can be the shallow ensembles of the Vietnamese study, or a neural recognizer that keeps spectral input and uses the aligned contour coordinates as an auxiliary target or pseudo-label source.
Why spectral input stays. We keep spectral input deliberately. In Japanese pitch-accent synthesis, bottleneck features from a speech recognizer beat raw pitch for capturing dialect accent (Cross-Dialect Text-to-Speech In Pitch-Accent Language Incorporating Multi-Dialect Phoneme-Level Bert). Removing non-pitch information from cepstral input also hurts Mandarin tone recognition (Tone Recognition Using Lifters and CTC). These results limit our design: explicit contours are used to estimate the dialect correspondence, not as the classifier's only input.
Secondary use as a dialect fingerprint. The fitted map $(\pi, \theta)$ between a reference inventory and an unknown speaker's inventory is itself a compact signature of that speaker's dialect. We will test whether it separates dialect pairs that a phone-based identifier confuses.
Baselines
The naive reference point is a tone classifier trained on the source dialect and applied unchanged to the target. On GlobalPhone Vietnamese this is the configuration reported at 50–57% cross-dialect accuracy, against roughly 78% within dialect and gender and 72.36% overall with the full feature set including the PaIntE contour-shape parameters, a pitch model with parameters such as peak height, alignment, and steepness (Cross-Gender and Cross-Dialect Tone Recognition for Vietnamese). We will re-run their feature set with Random Forest and Bagging on our own splits, so that every comparison shares speakers, syllables, and forced alignments.
The normalization-only ablation applies the same extrinsic speaker normalization and contour space but no transport or warp. This is the most important control. If dialect shift in tone were mostly a change in overall register, this ablation would capture it. Beating it is the condition for our claim to be meaningful.
A second ablation, shared-shift-only, keeps the transport step but forces $A_t=I$ and $c_t=0$. This isolates whether tone-specific shape changes matter beyond a uniform shift.
The strongest neural competitor is a Mandarin-style end-to-end tone recognizer: the cepstrogram-and-lifter model trained with connectionist temporal classification (CTC, a loss that trains sequence labels without frame alignments). It reports a 11.7% tone error rate on AISHELL-1 test, against 18.1% for a pitch-plus-MFCC recurrent baseline (Tone Recognition Using Lifters and CTC). We pair it with generic unsupervised adaptation, either iterative pseudo-labeling on target audio or feature-distribution matching, given the same unlabeled target hours and the same tuning budget as our method. For this model under dialect shift, and for self-supervised encoders on tone tasks, no reported result was found on Lacuna. We will establish those numbers ourselves.
Finally, a few-label oracle fine-tunes on 1%, 5%, and 20% of labeled target data, which yields a label-efficiency curve. A full target-supervised model gives the ceiling.
Beating these baselines means four things. We should be significantly above normalization-only and generic adaptation, by a margin that would justify the method; we take at least a third of the remaining cross-to-within-dialect gap as that margin. We should match or exceed what the few-label oracle achieves with a small amount of labeled data. All comparisons are tested with speaker-level bootstrap intervals.
Why not just…?
Why not just normalize pitch per speaker or per dialect? This is the most serious threat. The Wu study shows that good normalization by itself makes cross-dialect contours comparable. The evidence that it is not enough comes from shape, not level. Coarticulation and prosodic position change the curvature and the number of turning points of the Xiamen low rising tone (Effects of Tonal Coarticulation and Prosodic Positions on Tonal Contours of Low Rising Tones: In the Case of Xiamen Dialect). Taiwan Mandarin Tone 2 often surfaces with a dip resembling Tone 3 (Positional and phonotactic effects on the realization of taiwan Mandarin tone 2). Changde listeners weight register for one tone and slope for another (Tonal Perception in Changde Mandarin). None of these differences can be undone by a uniform rescaling. Still, the normalization-only ablation settles the question directly. If it comes within two points of our method, we will report that dialect tone shift is mostly register and drop the tone-specific warp.
Why not just fine-tune a large pretrained speech encoder on the target dialect? If labeled target data exist, this may well win, and the few-label oracle curve measures how few labels it needs. Our setting is unlabeled target audio, which is typical of spontaneous dialect speech without transcripts. The relevant evidence there is mixed. Whisper reaches 95.66% weighted F1 on in-domain Arabic dialect identification but drops to 62.74% on unseen TV drama (ADI-20: Arabic Dialect Identification dataset and models). Plain spectrogram tone recognizers sometimes fail to learn at all (Tone Recognition Using Lifters and CTC). Pseudo-labeling with a strong encoder is our generic-adaptation baseline. If it matches us, our remaining contribution is the readable correspondence map, and we will say so.
Why not just add contour-shape features such as PaIntE to the classifier? The Vietnamese study already did this. PaIntE parameters gave about 4 points of accuracy, yet cross-dialect accuracy still collapsed. Better features describe each dialect better but do not relate one dialect to the other. The source baseline above already includes these features, so any gain we report comes on top of them.
Proposed Experiments
The primary testbed is GlobalPhone Vietnamese. We run Northern-to-Southern and Southern-to-Northern adaptation, each crossed with gender so that dialect shift is not confounded with the 62–72% cross-gender drop. Southern Vietnamese is commonly described as merging two of the six orthographic tones; this is background knowledge, not from the file. If so, part of the reported collapse is not recognizer failure but a target category that cannot be recovered, and accuracy against orthographic tone labels is capped for that pair. We therefore report both raw accuracy and merger-aware accuracy, where the merged pair counts as one class.
A decisive check is whether the unbalanced transport, without being told, assigns those two source tones to a single target component, while the other tones receive one-to-one matches. A method that "improves" accuracy by forcing apart a genuine merger would be exposed by this check.
The second testbed is Mandarin. The source is AISHELL-1 (150 training hours, standard Putonghua), and the target is dialect-influenced Putonghua from the CL-CHILD corpus of 139 Changli-exposed children, if it is accessible. We measure tone accuracy against the corpus's on-target and off-target annotations. We also test whether the fitted per-tone warp for Tone 2 reproduces the documented blend: Tone 3 level and fall with Tone 2 slope. If it does, the method recovers a known phonetic finding without supervision.
The third, exploratory testbed is the 10-dialect Chinese Dialect Recognition Challenge. The two-stage phone-recognition system reaches 88.88% accuracy there but confuses Minnan with Hakka and Hebei with Shanxi (Two-Stage Training for Chinese Dialect Recognition). We will test whether concatenating our correspondence-map signature to its intermediate features reduces confusion on those specific pairs. Whether the challenge's phonetic annotations carry tone labels is unconfirmed, so this experiment uses only the label-free map.
The program succeeds if three things hold. On Vietnamese, the full method beats normalization-only and shared-shift-only by clear margins. The learned correspondences match phonetic descriptions: a merger where one is expected, and blend-shaped warps in the child data. Unlabeled adaptation matches a few-label fine-tuned encoder.
It fails informatively in two ways. If normalization-only suffices, dialect tone shift is mostly register, and we report that. If generic adaptation on neural features matches us across all splits, explicit contour geometry contributes interpretability but not accuracy, and we report that too.
References
- Cross-Gender and Cross-Dialect Tone Recognition for Vietnamese
- Pitch Target Realization in Putonghua Tone Production of Children from Dialect-Speaking Regions
- Dialect-geographical Acoustic-Tonetics: Five Disyllabic Tone Sandhi Patterns in Cognate Words from the Wu Dialects of ZhèJiāNg Province
- The role of creaky voice in Mandarin tone 2 and tone 3 perception
- Cross-Dialect Text-to-Speech In Pitch-Accent Language Incorporating Multi-Dialect Phoneme-Level Bert
- Tone Recognition Using Lifters and CTC
- Effects of Tonal Coarticulation and Prosodic Positions on Tonal Contours of Low Rising Tones: In the Case of Xiamen Dialect
- Positional and phonotactic effects on the realization of taiwan Mandarin tone 2
- Tonal Perception in Changde Mandarin
- ADI-20: Arabic Dialect Identification dataset and models
- Two-Stage Training for Chinese Dialect Recognition