Prosody-Derived Dialect Trees for Graceful Back-Off Under Domain Shift

Speech dialect classifiers fail badly when the recording conditions change. A self-supervised model that scores above 90% F1 on Arabic broadcast speech drops to about 5% F1 on dialogue from television dramas (On the Robustness of Arabic Speech Dialect Identification). A system that is wrong about the city could still be useful if it said "Levantine, unsure which country." That requires a tree of dialects, so the model can retreat from a leaf to an ancestor when it is uncertain. It also requires a way to decide when to retreat. Existing hierarchical dialect systems use administrative trees (Region → Country → City) taken from maps. Those trees encode politics, not how speech varies acoustically.

This note proposes building the dialect tree from speaker-normalized prosody: pitch contour shape, the timing of pitch peaks within syllables, rhythm and duration. Such features plausibly survive changes in microphone and genre better than the spectral detail self-supervised models rely on. The model would then make set-valued predictions restricted to nodes of that tree, with conformal calibration (a distribution-free way of choosing a confidence threshold from held-out data). The testable claim is that, under domain shift, predictions restricted to a prosody-derived tree keep more of their coverage at a coarser but correct level than predictions restricted to administrative, geographic, or confusion-derived trees. The tree is the object of study. The classifier stays fixed.

Core Claim

The choice of hierarchy is a modeling decision with measurable out-of-domain consequences. When a dialect classifier is forced to answer with a single tree node, the tree that best preserves correctness under shift has grouping structure that matches the features least affected by the shift. We hypothesize that for spoken dialects these are slow prosodic properties: contour shape classes, peak alignment, and vocalic proportion. Fine spectral cues are more affected by channel. If this holds, a prosody tree built once on in-domain data should transfer: its internal nodes stay "pure" on drama speech even when leaf-level accuracy collapses.

Relation to Prior Work

Hierarchy has already been tried for dialect identification, but only as administrative trees over text. Hierarchical Aggregation of Dialectal Data for Arabic Dialect Identification trains character and word language models at each level of a Region → Country → City tree and reaches 68.13% city accuracy on MADAR, against 67.69% without the region level. Hierarchical Deep Learning for Arabic Dialect Identification cascades a group classifier into per-group classifiers and gets 0.58 F1, against 0.56 for a flat network. Its errors at the first level cannot be undone at the second, and a flat Naive Bayes plus Random Forest vote beats both at 63.02% F1. These papers motivate the proposal: hierarchy helps a little. They also constrain it: hard cascades are brittle, so we use soft, calibrated back-off instead. Neither paper works with audio, and neither asks whether the tree itself is right.

The hierarchy-learning literature supplies our methods. Making Better Mistakes contributes two things. One is the measure of mistake severity, the tree distance between the predicted and true class. The other is hierarchical cross-entropy, which factorizes the loss along the path to the root. The two conformal hierarchical papers (Conformal Prediction in Hierarchical Classification; with Constrained Representation Complexity) restrict prediction sets to tree nodes while keeping coverage guarantees. We reuse that machinery. All of this work treats the tree as given, and we vary it.

The prosody literature tells us which features to use for building the tree. Using prosody and phonotactics in Arabic dialect identification defines pseudo-syllable features: mean pitch, pitch slope, peak alignment, pitch range, duration, vocalic percentage, intensity, and speaking rate. Modeled as sequences, these features alone reach 64.33% on four dialects, against 54.83% as global statistics. Tonal alignment in three varieties of hiberno-English shows that varieties differ in whether peak alignment is stable or drifts with the surrounding unstressed syllables. That is a grouping property invisible to categorical prosodic transcription. Data-driven approaches to pitch modelling in two Mexican Spanish ethnolects clusters speaker-z-scored pitch contours into shape classes whose frequencies differ by variety. We turn these findings into a distance between dialects. Audio-Based Classification and Geographic Regression of Austrian Dialects is the cautionary case: federal-state labels such as Burgenland classify poorly because several dialect groups cut across them. This is direct evidence that administrative trees misgroup speech.

Research Program

We first build candidate trees over the 17 ADI-17 dialects, all from in-domain training audio. The prosody tree comes from per-dialect distributions of the pseudo-syllable features above, plus occupancy of k-means contour shape classes computed on time-normalized, speaker-z-scored pitch. We take energy distances between dialect distributions and apply average-linkage agglomerative clustering. The competing trees are four. The administrative tree is Region → Country. The geographic tree comes from clustering capital-city coordinates. The confusion tree comes from clustering the symmetrized confusion matrix of the in-domain classifier. The fourth is a random tree with the same shape as the prosody tree. All trees are binarized or matched in depth where possible, so that node counts are comparable.

Next we fix one classifier: the HuBERT model fine-tuned on ADI-17 from the robustness paper's setup, retrained by us to reproduce its numbers. Leaf probabilities $p(y\mid x)$ induce a node probability $P(v\mid x)=\sum_{y\in \mathrm{leaves}(v)} p(y\mid x)$ for every node $v$. For a given tree $T$, the prediction is the deepest node that holds enough mass:

$$\hat v_T(x)=\text{deepest } v\in T \text{ such that } P(v\mid x)\ge \tau_T,$$

where $\tau_T$ is chosen on a calibration split so that the true leaf lies under $\hat v_T(x)$ with frequency $1-\epsilon$ (we use $\epsilon=0.1$). Because the classifier is shared, any difference between trees comes from the grouping alone.

Conformal guarantees assume calibration and test data come from the same distribution, and drama speech breaks that. We therefore report two regimes. In the first, the threshold is calibrated in-domain and we measure the coverage lost out-of-domain, which is the quantity of interest. In the second, the threshold is recalibrated on a small labeled slice of drama audio, and we compare how specific (deep) the resulting predictions are. In a secondary arm, we train the classifier with hierarchical cross-entropy on each tree. This asks whether the tree should shape training or only decoding.

Baselines

The naive reference is the flat HuBERT classifier with a top-1 leaf prediction, and also with flat conformal sets of arbitrary leaves. The robustness paper reports 92.12% F1 for it on ADI-17 and about 5% F1 on YouTube Dramas. The flat-set variant matters because unconstrained leaf sets can be smaller than any tree node. The tree constraint must justify itself by returning a single named group, while costing little in set size, at matched coverage.

The strongest existing method for out-of-domain accuracy is self-training on unlabeled drama audio, which reaches 45.10% F1 according to On the Robustness of Arabic Speech Dialect Identification. We run it with the same backbone and the same unlabeled pool. Prosody-tree back-off is orthogonal to it, so we also apply back-off on top of the self-trained model. Beating it means that, for the same model, the prosody tree gives higher out-of-domain coverage at equal average node depth than the administrative tree does.

The strongest structural competitor is the confusion-derived tree. It is built from exactly what the classifier finds hard, so it is the natural fit for decoding. The administrative tree corresponds to the MADAR-style hierarchy of the two Arabic text papers, ported to audio. The geographic tree stands in for the continuous geography of the Austrian regression work. The random tree of identical shape is the ablation that separates "any hierarchy helps back-off" from "this hierarchy helps." Every tree uses the same classifier, the same calibration split, and the same $\epsilon$, and none involves tuning on drama labels except in the explicit recalibration regime. The proposal succeeds if, at in-domain calibrated $\epsilon=0.1$, the prosody tree loses significantly less drama coverage than both the confusion and administrative trees at matched mean depth. We test significance with a speaker-level bootstrap.

Why not just…?

Why not just improve the classifier with domain adaptation? Self-training is a real advance, but at 45.10% F1 the leaf predictions are still wrong more often than right. A calibrated coarse answer remains needed, and the two approaches stack. We report back-off on both the base and the self-trained model. If self-training closes most of the gap, the benefit of the prosody tree should shrink, and we will report that.

Why not just use the confusion matrix to build the tree? This is the most serious threat, because the confusion tree is tuned to the classifier's in-domain failure modes. Our hypothesis is that in-domain confusions reflect channel-dependent cues and reorganize under shift. We test this directly. Using a few labeled drama clips, we compute the confusion tree and the prosody tree separately in-domain and on drama audio, and measure how much each changes between the two (a tree-distance comparison). If the confusion tree is equally stable and decodes equally well, the prosodic premise fails. In that case the paper's contribution becomes the negative result plus the evaluation protocol.

Why not just regress geographic coordinates or use distance-weighted soft labels? The Austrian work reaches 66.7 km mean error with coordinate regression, but a point estimate gives no calibrated statement about which group a sample belongs to. Their state-level failures in Burgenland show that geography and dialect grouping come apart. The geographic tree baseline measures the effect directly.

Why not just output flat conformal leaf sets? They may well be smaller. The cost is that a set such as {Jordan, Sudan, Kuwait} is not a usable answer. We report set size for flat sets alongside node size, so readers can see what the single-node constraint costs.

Proposed Experiments

The main experiment calibrates on ADI-17 and tests on the ADI-17 test set, on ADI-5 without MSA (80.36% F1 reported for HuBERT-17 transfer), and on YouTube Dramas. The metrics are coverage, mean node depth, mistake severity in the style of Making Better Mistakes, and leaf F1. We have assumed the drama audio carries country-level labels; this needs to be checked before the study starts. MSA does not sit naturally under any regional node, so in every tree we make it a child of the root, which keeps the comparison fair.

The diagnostic experiment measures the shift in per-dialect feature distributions between broadcast and drama audio, separately for the prosodic features and for HuBERT embeddings. This checks whether the premise of channel robustness holds before any claim about trees is made.

The replication runs on the Austrian corpus. We compare a prosody tree over the 108 locations with the federal-state tree and the dialect-group tree, using cross-speaker splits. The reported 72.7% dialect-group accuracy for XLSR anchors the leaf level.

The decisive check combines the diagnostic and the main experiment. If prosodic distributions do not shift less than embeddings, or if the random tree of the same shape matches the prosody tree under shift, then the claim is false. Either outcome is informative about what hierarchy constraints actually contribute.