Catching Unseen Phones in a Reconstruction Bottleneck

A speech recognizer trained on a few dozen languages knows a fixed set of speech sounds. When it hears a language that uses a sound outside that set, such as an ejective, a click, or an unusual vowel, it does not report that the sound is unfamiliar. It writes down the closest sound it already knows. Zero-Shot Cross-Lingual Phonetic Recognition with External Language Embedding names this collapse onto the nearest known phone as a main failure mode of zero-shot transfer. Even the system that handles it best, Allophant, builds phones out of shared articulatory parts and still decodes only into inventories listed in a curated table. We propose to detect these out-of-inventory phones directly from the acoustics, so that a field linguist or a downstream system is told "this segment is something the model has not heard before" and is not handed a confident wrong symbol.

The central bet concerns where that evidence survives. Work on cross-lingual articulatory inversion (Exploring Self-Supervised Speech Representations for Cross-lingual Acoustic-to-Articulatory Inversion) found that fine-tuning a self-supervised speech model for recognition destroys fine articulatory detail. We hypothesize that the same thing hides novel phones: recognition training pulls an unseen sound onto a known category, so a classifier's own confidence and features cannot reveal it. An autoencoder is a network that squeezes its input through a narrow layer and must rebuild the input from that layer. Because it has to rebuild the acoustics, it is forced to keep sub-phonemic detail. A distance-based novelty score in that narrow layer should therefore separate unseen phones from known ones far better than the recognizer can.

Core Claim

We claim the following. An autoencoder trained on intermediate-layer self-supervised speech features, with a reconstruction loss and a light articulatory-attribute head, produces a low-dimensional space in which out-of-inventory phones in unseen languages can be detected by neighborhood geometry. Its detection is substantially better than four alternatives: the recognizer's softmax confidence, distances in the recognizer's fine-tuned features, distances in the raw frozen self-supervised features, and a linear projection (PCA) of those features to the same size.

What would be new is the task itself: phone-level novelty detection as a first-class output of cross-lingual phone recognition. The second novelty is the controlled evidence for which ingredient makes it possible. The candidates are compression, nonlinearity, reconstruction, or attribute guidance.

Positioning Against Prior Work

Two lines of work supply the method.

  • Tree-based Ensemble Learning for Out-of-distribution Detection (TOOD) projects images and text into an autoencoder latent space, where distances stop concentrating. It then scores novelty by how much tree-ensemble leaf assignments disagree, reporting 99.1% AUROC on CIFAR-10 and 99.7% on 20Newsgroups. We reuse its "compress, then score by geometry" recipe. However, it was never applied to speech or to sub-utterance units. Its $O(M^2)$ pairwise cost also forces us to use approximate nearest neighbors at frame scale.
  • Learning Robust Speech Representation with an Articulatory-Regularized Variational Autoencoder shows that shaping an autoencoder's latent space toward vocal-tract configurations helps speech modeling. It uses physical articulography recordings from single French speakers. We replace those recordings with the attribute tables used by Allophant, so the guidance is available for many languages.

Three lines of work motivate the gap.

  • Gao et al. inject language-family and inventory embeddings and mask decoding to the target inventory. That mask makes out-of-inventory phones structurally unrecognizable.
  • Allophant composes phones from attributes and gains 2.63 percentage points of phone error rate over its shared baseline on the UCLA Phonetic Corpus. It has no bottleneck, no reconstruction objective, and no way of abstaining.
  • The inversion study supplies the key mechanism and constraint: articulatory information peaks around two-thirds of the way up the self-supervised model, and recognition fine-tuning erodes it. We take our input features from that depth.

Three lines of work using bottlenecks constrain the design.

We deliberately leave tone aside. Allophant's attribute system excludes it, and folding pitch in would confound the segmental question.

Research Program

The encoder input is a frame from a frozen XLS-R layer at roughly two-thirds depth. The autoencoder maps it to a code $z$ of dimension $d$ (we sweep $d$ from 16 to 128) and rebuilds the input frame together with a short context window. Training minimizes

$$\mathcal{L} = \|x - \hat{x}\|^2 + \lambda_a \sum_k \mathrm{CE}(a_k, h_k(z)) - \lambda_s\, \mathrm{CE}(s, q(z)).$$

The symbols are:

  • $x$ and $\hat{x}$ are the input and reconstructed features.
  • $a_k$ is the $k$-th articulatory attribute label of the aligned phone (for example voicing, place, or manner), taken from Allophoible.
  • $h_k$ is a small classifier for attribute $k$.
  • $s$ is the speaker label, and $q$ is a speaker classifier trained through gradient reversal, following Janbakhshi and Kodrasi.
  • $\lambda_a$ and $\lambda_s$ are weights.

The attribute head is kept weak on purpose. It should organize the code space so that distances reflect articulation, without collapsing it onto known categories.

Training data are the 34 Common Voice languages used by Allophant, with phone labels and alignments produced the same way that work produces them. The novelty score of a segment is its mean distance to the $k$ nearest training codes, computed over the segment's frames. We also report a TOOD-style tree-disagreement score as a secondary detector.

Flagged segments get a second use. We decode their attribute bundle from $h_k(z)$ and ask whether the predicted voicing, place, and manner match the reference description of the unseen phone. This tests whether the space can describe a new sound, not merely flag it.

Baselines

The naive reference point is the Allophant recognizer itself, scoring each segment by maximum softmax probability or by posterior entropy. We use the publicly described configuration, retrained on the same Common Voice data with the same alignments. This is what a practitioner gets for free. Its expected weakness is the collapse mechanism we hypothesize: over-confident predictions of the nearest known phone. On 20Newsgroups, TOOD reports that max-softmax reaches only about 65% AUROC, against 99.7% for latent geometry. That result is suggestive, but it comes from a different domain.

The strongest competitor is $k$-nearest-neighbor distance computed directly on the frozen XLS-R features at the same intermediate layer, using cosine distance and no learned compression. It already sits where the inversion study found the most articulatory information, and it needs no training. If it matches us, the autoencoder adds nothing. No reported result was found on Lacuna for out-of-inventory phone detection with any method, so the comparison is ours to establish. The nearest reported numbers are Allophant's 2.63 pp phone error rate gain on UCLA and Gao et al.'s 4% to 8% absolute token error reduction, both measured on recognition, not detection.

Ablations isolating the key ingredients come next. A PCA projection of the same features to the same $d$ tests whether compression alone suffices. The same encoder trained with attribute and speaker losses but no reconstruction term tests whether reconstruction is the active ingredient. This is the analogue of the classification-only bottleneck of Fu et al. Removing the attribute head tests whether pure reconstruction already organizes the space. Removing the speaker adversary tests whether speaker leakage produces false alarms. kNN on the fine-tuned recognizer's penultimate layer tests the "fine-tuning erases novelty" hypothesis directly.

Fairness controls apply to every method. All methods use the same training frames as the neighbor reference set, the same $k$ and neighbor index, and the same segmentation. Each gets an equal hyperparameter search budget tuned on held-out Common Voice languages, never on UCLA. Beating the strongest baseline means a statistically reliable AUROC gain, measured by a bootstrap over languages, on the within-language contrast described below. A gain on the raw pooled evaluation alone does not count.

Why not just…?

Why not just threshold the recognizer's confidence? It is the cheapest option and must be reported. The mechanism argues against it: recognition training rewards mapping every sound onto a known category, and the inversion study shows such training erodes continuous articulatory detail. The decisive check is the fine-tuned-penultimate-layer baseline. If its neighbor distances are as good as our bottleneck's, the "fine-tuning erases novelty" story is wrong and confidence-style methods may be competitive after all.

Why not just use nearest-neighbor distance on frozen self-supervised features? This is the real threat. Intermediate XLS-R layers keep articulatory information without any added training, and cosine distance in about 1,000 dimensions may not suffer badly from distance concentration in practice. Our argument rests on two points. First, compression plus attribute guidance should down-weight dimensions that encode speaker and channel. Second, TOOD found that compressed spaces made geometric scoring work where raw spaces did not. This is an empirical question, and the head-to-head comparison settles it. If frozen kNN wins or ties, we will report that the autoencoder is unnecessary and that the contribution is the task, the benchmark, and the layer choice.

Why not just use PCA? If linear compression matches the autoencoder at equal $d$, then nonlinearity and reconstruction are not doing the work. We would recommend PCA as the simpler tool. The no-reconstruction ablation separates "any compression helps" from "reconstruction preserves novelty-relevant detail," which is the claim we actually care about.

Proposed Experiments

Evaluation set. The main test is the UCLA Phonetic Corpus, as used by Allophant: 95 languages unseen in training. For each language, we split reference phones into those inside the 34-language training inventory and those outside it.

Within-language contrast. The decisive design choice is to measure detection within each unseen language. Known and unknown phones then share the same speakers, microphones, and recording sessions. A detector that merely notices a new language or channel scores near chance on this contrast. We report this within-language AUROC as the primary metric, alongside the easier pooled AUROC.

Segment boundaries. Segment boundaries come from forced alignment of the reference transcription, with unknown phones mapped to a placeholder unit. Because this alignment is noisy, we also report utterance-level detection (does this word contain an unseen phone?), which needs no boundaries at all.

Controlled check. We retrain every system with one natural class removed, for instance all ejectives or all retroflexes, by dropping the Common Voice languages that rely on it. We then test on held-out Common Voice data where ground-truth alignment is cleaner. This tests whether detection follows the held-out class rather than accidents of the UCLA recordings.

Secondary measurements. Three further measurements follow:

  • Attribute accuracy on flagged segments, compared against Allophant's own attribute classifiers.
  • Phone error rate when flagged segments are emitted as attribute bundles instead of nearest phones, scored with a feature-weighted edit distance.
  • A sweep over bottleneck size $d$, which shows the trade-off between too narrow (losing distinctions) and too wide (letting speaker identity through, measured by a speaker probe on $z$).

What would kill the claim. The idea fails if, on the within-language contrast, the frozen XLS-R kNN baseline or PCA equals the full model, or if removing the reconstruction loss costs nothing. It survives only if reconstruction is shown to be the ingredient that keeps unseen phones from collapsing onto known ones.