Speech recognizers for low-resource and unwritten languages usually learn from a handful of well-resourced languages. They then have to transcribe sounds they were never taught to name: a retroflex stop, an ejective, a vowel that no training language has. Several methods claim to handle this "inventory mismatch" between training and target sound sets. Some map unseen sounds to the nearest known sound by articulatory features. Some mask the output to the target language's known inventory. Some route predictions through a learned graph of phone-to-phoneme rules. Some compose phoneme representations from articulatory attributes. Each was evaluated with a different speech encoder, different training languages, different test languages and a different error metric. As a result, nobody can say which way of handling mismatched inventories actually helps, or for which kind of sound.
We propose to answer that question under controlled conditions. We hold one pretrained speech encoder and one training set fixed, and treat the inventory-mapping step as a swappable module behind a single interface. We then score every system with a metric that breaks errors down by how far each reference sound lies from anything in the training inventory. The central experiment is a controlled withholding test. We deliberately relabel specific training contrasts so the model never sees a label for certain sounds that are still present in the audio. Because we know exactly which sounds were withheld, we can measure how each mapping module recovers them rather than inferring it after the fact. The contribution is a scientific finding about where mapping modules differ, with modular software as the tool that makes the comparison fair. The software is not the goal.
Core Claim
We claim that the reported differences between zero-shot phone recognition methods are mostly confounded by encoder, data and metric choices. Once these are fixed, the real differences between mapping modules concentrate in a narrow band of sounds: those one to two articulatory features away from the training inventory. Aggregate Phone Error Rate (PER, edit distance over phone symbols) hides this band. Phone Feature Error Rate (PFER, edit distance weighted by articulatory feature disagreement) blurs it. A mismatch-stratified score makes it visible.
For each reference phone $p$ in a test utterance, we define its mismatch distance as
$$d(p) = \min_{q \in \mathcal{I}_{\text{train}}} \mathrm{H}\big(f(p), f(q)\big),$$
where $\mathcal{I}_{\text{train}}$ is the set of phones labeled anywhere in training, $f(\cdot)$ is the 24-dimensional PanPhon articulatory feature vector, and $\mathrm{H}$ is Hamming distance. After standard alignment of hypothesis and reference, we attribute substitution and deletion errors to the reference phone's bucket: $d=0$ (seen), $d=1$, $d=2$, and $d\ge 3$. Insertions are reported separately, since they have no reference phone. The prediction is falsifiable. If module rankings are the same in every bucket and match the aggregate ranking, stratification adds nothing and we will say so.
Relation to Prior Work
Five papers motivate the question, because each reports gains on its own setup and none can be compared directly with the others.
Simple and Effective Zero-shot Cross-lingual Phoneme Recognition fine-tunes all of XLSR-53 with a standard sequence loss (CTC) and maps unseen phonemes to the nearest training phoneme by articulatory Hamming distance. It argues that specialized architecture is unnecessary. Zero-Shot Cross-Lingual Phonetic Recognition with External Language Embedding conditions an English-pretrained wav2vec 2.0 on language-family and inventory vectors and masks decoding to the target inventory. Differentiable Allophone Graphs for Language-Universal Speech Recognition inserts a trainable weighted finite-state transducer, a graph that converts universal phones into language-specific phonemes. Allophant builds phoneme embeddings from articulatory attributes and adds attribute classifiers. Universal Automatic Phonetic Transcription into the International Phonetic Alphabet trains an open 6,487-phone output layer on nine hours of cleaned data and introduces PFER-based evaluation with a human agreement ceiling. Each uses a different encoder, training set and test languages. We reuse their mapping mechanisms as modules and discard everything else that differs between them.
Two results constrain the design. First, Allophant's authors found that an explicit intermediate allophone layer overfit compared with a simple shared layer. More modularity inside the model is therefore not automatically better, so we keep the swappable boundary narrow: one interface between encoder outputs and output symbols. Second, the monolithic result of Xu et al. is a live threat to the whole premise. If full fine-tuning plus nearest mapping ties everything else in every bucket, that is a publishable negative result, and our design must be able to show it.
The methodology comes from standardized-evaluation work outside speech. BARS showed that claimed recommender gains vanished when baselines were tuned under one protocol. NerfBaselines showed that preprocessing details flipped leaderboard rankings. A Sober Look at the Unsupervised Learning of Disentangled Representations traced method gains to seeds and tuning. The separation of data iteration, inference, post-processing and metric computation follows OmniEvaluator. OmniEvaluator's audio coverage stops at coarse tasks like LibriSpeech, and its learned judge reads only text, so it offers no phone-level or mismatch-aware scoring. We borrow its staged structure, not its metrics.
For test data, Phonetic Segmentation of the UCLA Phonetics Lab Archive (VoxAngeles) supplies audited IPA transcriptions and alignments for 95 languages. That paper also shows that off-the-shelf forced alignment fails on these languages. This is why we rely on its audited labels rather than generating our own.
Research Program
We will build a small harness with three fixed stages and one swappable stage. The fixed stages are data loading with frozen language splits, a shared encoder (XLS-R through fairseq), and a metric engine that computes PER, PFER and the bucketed scores from IPA strings normalized through PanPhon. The swappable stage is the mapping module. It receives frame-level encoder outputs and a declared amount of target-language knowledge, and it emits an IPA string.
We will implement six modules. The first is a plain training-inventory CTC head. The second is that head plus nearest-feature remapping (Xu et al.). The third is the head with language conditioning and a target-inventory mask (Gao et al.). The fourth is a differentiable allophone graph with the universal constraint, implemented in GTN (Yan et al.). The fifth is compositional attribute embeddings with a multitask attribute loss (Allophant). The sixth is an open PanPhon-vocabulary head (Taguchi et al.).
Each module must declare what it knows about the target language: nothing, the phoneme inventory, or a few labeled utterances. We report results within these knowledge tiers. The inventory mask, for example, uses information the open-vocabulary head does not. Without the tiers, "better" would partly mean "told more."
The main training regime follows the clean, small setting of Taguchi et al.: seven Common Voice languages at roughly 1,000 verified utterances each. We will run a second, larger regime in the spirit of Allophant's 34 Common Voice languages to check whether conclusions survive scale. Tone is excluded from all scoring, because every method compared omits it. We state this as a limitation, not a solved problem.
The withholding experiment works as follows. We choose contrasts that are present in the training audio, such as retroflex versus dental or a rounded front vowel. In the training labels we merge each withheld sound into its nearest neighbor. The model therefore hears the sound but never receives its label, and $d(p)$ for that sound becomes known and controlled. We then test on VoxAngeles languages where those sounds occur, and measure per module how often the withheld sound is emitted correctly versus collapsed into its training neighbor. This makes the "category collapse" described in cross-lingual work directly measurable. Of the modules, only the open-vocabulary head, the attribute-composition head and nearest-feature remapping can in principle emit an unlabeled sound, so the experiment also tests whether that ability holds up in practice.
Baselines
The do-nothing reference is the plain CTC head over the training inventory, with no remapping and no target knowledge. By construction it scores zero correct in every $d\ge1$ bucket. This mirrors the 100% error on unseen phonemes reported for the baseline in Towards Zero-Shot Learning for Automatic Phonemic Transcription, where the attribute-based model reached 89.8% PER on those phonemes. Every other module must beat it in the $d\ge1$ buckets without losing more than a bootstrap confidence interval at $d=0$.
The strongest existing system in the no-target-knowledge tier is the open-vocabulary approach of Taguchi et al. On the four unseen Common Voice languages (Luganda, Upper Sorbian, Hakha Chin, Tatar), it reports 21.2% PFER, against 22.4% for Wav2Vec2Phoneme and 34.2% for Allosaurus, with human inter-annotator agreement at 19.6% PFER. We will first reproduce this setting in our harness and accept the harness only if we land within about one PFER point. Taguchi et al.'s own released pipeline serves as the reproduction check. A module beats this baseline only if it wins on aggregate PFER on those four languages and in at least the $d=1$ and $d=2$ buckets on VoxAngeles, with non-overlapping bootstrap intervals across three seeds.
The strongest system that uses more target knowledge is inventory masking with language conditioning (Gao et al.), which reports a 4% to 8% absolute reduction in phonetic token error rate over multilingual baselines on Cantonese, Lao, Croatian, Spanish and Polish. It competes only within the inventory-known tier. The simplest strong competitor overall is full fine-tuning with nearest-feature remapping. It reports 63.6% phonetic token error rate on zero-shot Cantonese in BABEL, against a prior 73.1%. For the allophone graph, the reference figure is 80.2% PER on Tusom and Inuktitut, against 94.0% for a static mapping matrix. For Allophant, only relative gains are available: 2.63 points of PER over its shared-layer baseline on the UCLA corpus. We found no reported absolute PER on Lacuna for Allophant's full model on that corpus.
All modules share the same encoder checkpoint, training utterances, number of optimizer steps and hyperparameter search budget, set at twenty trials per module on a held-out development language. This follows the BARS lesson that untuned baselines manufacture gains.
The ablation that isolates our key ingredient is to compute everything twice: once with aggregate PER/PFER only, and once with the bucketed score. The question is whether the bucketed score changes any conclusion about which module to use.
Why not just…?
Why not just run each paper's released code on one common test set? This would fix the test set but still leave encoder, pretraining data and training languages varying with each method. Any ranking would still mix mapping effects with backbone effects. We will run it anyway, as a reference row. The gap between that ranking and our fixed-encoder ranking is itself a measurement of how much published comparisons are confounded.
Why not just report PFER, as Taguchi et al. already do? PFER already credits near-miss substitutions, so it partly captures mismatch. It is the most serious threat to our claim. However, PFER averages over all phones, and in most test languages the $d=0$ phones dominate. A module that is much better on rare distant sounds and slightly worse on common seen sounds can therefore tie on PFER. The decisive test is whether we find any pair of modules whose PFER ranking reverses within a bucket. If no such reversal appears on VoxAngeles in either training regime, bucketing is redundant and we will recommend plain PFER.
Why not just fine-tune a larger encoder end to end and skip mapping modules entirely? Xu et al.'s result suggests this may be enough. Our design tests it directly: full fine-tuning with nearest remapping is one of the six modules, and the withholding experiment shows whether it can emit a sound it heard but never saw labeled. If it matches the structured modules in every bucket, the conclusion is that mismatch handling belongs in data and encoder choice, not architecture. That would be a useful outcome for the field.
Proposed Experiments
The first experiment tests ranking stability. We train all six modules on the clean seven-language regime with three seeds and evaluate on the Taguchi unseen languages, on VoxAngeles (95 languages across 21 families), and, where licensing allows, on BABEL Cantonese and Lao. We compare the fixed-encoder ranking with the ranking implied by published numbers and with the released-code ranking. We predict the fixed-encoder spread on aggregate PFER will be much smaller than the published spread.
The second experiment tests whether stratification matters. We compute bucketed substitution, deletion and insertion rates per module and test for within-bucket ranking reversals against aggregate PFER, using bootstrap intervals over languages. We predict that module differences concentrate at $d=1$ and $d=2$, with the dominant remaining errors at $d\ge3$ being deletions, consistent with the persistent insertion and deletion errors that UPM and AlloGraph analyses report.
The third experiment is controlled withholding. For four to six withheld contrasts, we measure per module the rate of correct emission, of collapse into the training neighbor, and of deletion on VoxAngeles tokens of the withheld sound. We predict that attribute-composition and open-vocabulary heads recover some withheld sounds, while graph and mask modules cannot emit them by construction. The empirical question is whether recovery exceeds a few percent in practice.
The fourth experiment is a scale check. We repeat the first and third experiments in the larger Common Voice regime to see whether rankings and recovery rates hold as data grows, which bears on Taguchi et al.'s finding that clean, diverse data plateaus early.
The note stands or falls on the second and third experiments. If rankings reverse within buckets and withheld-sound recovery separates the modules, we will have shown that aggregate metrics have been answering the wrong question about inventory mismatch. If neither happens, we will have shown that the field's mapping machinery matters much less than its evaluation choices.
References
- Simple and Effective Zero-shot Cross-lingual Phoneme Recognition
- Zero-Shot Cross-Lingual Phonetic Recognition with External Language Embedding
- Differentiable Allophone Graphs for Language-Universal Speech Recognition
- Allophant: Cross-lingual Phoneme Recognition with Articulatory Attributes
- Universal Automatic Phonetic Transcription into the International Phonetic Alphabet
- BARS: Towards Open Benchmarking for Recommender Systems
- NerfBaselines: Consistent and Reproducible Evaluation of Novel View Synthesis Methods
- A Sober Look at the Unsupervised Learning of Disentangled Representations and their Evaluation
- A Composable Evaluation System for Reproducible Omni-Modal Foundation Model Evaluation
- Phonetic Segmentation of the UCLA Phonetics Lab Archive
- Towards Zero-Shot Learning for Automatic Phonemic Transcription