Can a Phone Confusion Matrix Stand In for Private Speech Data?

Many of the most interesting results in spoken-language systems cannot be checked by anyone else. Voice query auto-completion was studied on 1.1 million smart-TV queries that will never be released. Personalized recognition of names was measured on in-house contact catalogs. Clinical speech screening runs on recordings protected by privacy law. This note proposes a narrow way through. The data owner does not release audio or transcripts. It releases one aggregate statistic: a phone confusion matrix, meaning a table of how often its own speech recognizer turns each speech sound into each other sound. Outside researchers then use this table to corrupt public text, producing an open stand-in benchmark that carries that recognizer's characteristic errors, such as hearing "who" when the user said "Hulu."

The question that decides whether this is worth doing has not been asked in the literature: does such a stand-in benchmark reach the same scientific conclusions as the private data? A proxy that matches overall error rate but reorders which methods win is worse than useless. We propose to measure this directly, using "rank fidelity," the agreement between method rankings on simulated and on real recognizer output. We test whether errors shaped by acoustic distance preserve those rankings when cheaper corruption schemes do not.

Core Claim

Recognizer errors are structured. A recognizer that mishears "bat" is far more likely to output "pat" than "rug," because /b/ and /p/ differ only in voicing. A confusion matrix captures most of this structure in a few thousand numbers that contain little personal information. Our claim is that text corrupted through a recognizer-specific, acoustically weighted confusion model preserves the relative ordering of downstream methods on spoken-language-understanding tasks. Here "preserves" means rank correlation with real-transcript results of at least 0.8, and an estimated improvement of each robust method over the baseline that falls within half of its true improvement. We further claim that these properties fail for corruption that ignores acoustics, even when its overall word error rate is matched. If random character noise at matched error rate ranks methods equally well, the acoustic-distance ingredient is unnecessary and the claim is false.

Relation to Prior Work

The method comes from Discriminative language modeling using simulated ASR errors. That work maps words to phones, passes them through a confusion transducer, and maps the result back to words. The confusion transducer is built either from recognizer logs or from distances between phone hidden Markov models. Error-free text corrupted this way trained discriminative n-gram language models as well as real audio lattices did (48.5% versus 48.4% word error rate on Switchboard). That paper used simulation to train a model and judged success by a single error rate. It never asked whether simulated data supports comparisons between methods. Its word-level back-projection also cannot produce out-of-vocabulary names, which are exactly the entities that fail most in spoken understanding.

Two works motivate the reproducibility side. Voice Query Auto Completion reports an 18% relative gain in mean reciprocal rank on the private EntSys1M corpus, a result no one can re-test. Retrieve and Copy pairs private name catalogs with public VoxPopuli, but its catalogs stay closed. It also handles sound-alike names only implicitly, by mining hard negatives in an embedding space.

The target application comes from AR-NLU. It trains text encoders to be robust to recognizer noise on SLURP and SLUE, using transcripts from a commercial recognizer. It therefore inherits a vendor-drift reproducibility problem and treats errors as arbitrary string corruption.

Two findings constrain the design. First, Phoneme confusions in human and automatic speech recognition shows that machine confusions, especially among vowels, differ systematically from human ones. A matrix must therefore come from the target recognizer, not from linguistic intuition. Second, Exploiting phoneme similarities in hybrid HMM-ANN keyword spotting found that unfiltered acoustic expansion floods results with real competing words, lowering figure-of-merit from 64.30% to 61.9%. This warns that simulated errors must be weighted by word frequency, not generated uniformly over all sound-alike strings.

Research Program

We build a simulator in the style of Jyothi and Fosler-Lussier, with three changes.

First, the confusion table includes insertions and deletions. It is estimated only from aggregate phone-alignment counts that a data owner could compute on its own development set.

Second, the back-projection allows open-vocabulary output. Confusable phone strings either map to in-vocabulary words, weighted by a language-model prior that plays the filtering role of Pinto et al.'s language confusion matrix, or are spelled out through a phone-to-spelling model. This lets names come out as plausible misspellings rather than being forced into dictionary words.

Third, an optional context-dependent version conditions each substitution on the neighboring phones. This addresses the known blindness of flat matrices to how sounds blend into one another.

Because a confusion matrix is still a release, we add calibrated noise to its counts at several strengths. We report how rank fidelity degrades as noise increases. This gives data owners a concrete privacy–fidelity curve, though we do not claim a formal privacy guarantee for the matrix itself.

To evaluate rank fidelity we need ground truth, so we stage the private setting with public data. Several open recognizers, for example fine-tuned versions of the SpeechT5 codebase and other open acoustic models of varying quality, each play the "private recognizer." Each transcribes the SLURP and SLUE-VoxPopuli audio, and these real transcripts are the hidden truth. Each recognizer's confusion matrix is estimated on a held-out development split only. The simulator then corrupts the gold transcripts of the training and test splits.

We fix a pool of about eight understanding methods:

  • a BERT intent classifier trained on gold text;
  • the same classifier trained on noisy text;
  • the AR-NLU triplet-loss and KL-divergence variants;
  • BINDER and AR-BINDER for entity recognition;
  • two augmentation-based variants.

Each method is run end to end twice: once in the "real world," trained and tested with real recognizer transcripts wherever it needs noisy inputs, and once in the "proxy world," trained and tested with simulated ones. We then compare method rankings across the two worlds, along with the size of each method's gain over the baseline.

Baselines

The naive reference is no noise at all: evaluating every method on gold transcripts. SLURP already shows that this overstates performance. Action accuracy is 86.99% on perfect text versus 81.42% on recognizer output, and entity SLU-F1 is 78.19 versus 70.84 (SLURP). Its rank fidelity is the floor any proxy must clear.

The second baseline is orthographic noise. Random character and word insertions, deletions, and substitutions are tuned so that word error rate matches the target recognizer exactly. This is the decisive ablation, because it isolates whether acoustic shape, rather than error amount, carries the ranking signal.

The third baseline is a flat, uniform phone-level confusion model at matched error rate. It is acoustic in form but carries no recognizer-specific information, and so isolates the value of the released matrix.

The fourth, and the strongest competitor, is cross-recognizer substitution: running a different open recognizer on the same public audio and treating its transcripts as the proxy. It produces real acoustic errors with no simulation at all.

All proxies receive identical text splits, the same matched error rate, and the same number of method runs. All methods get the same hyperparameter budget in both worlds.

As a sanity anchor, our simulator must reproduce the AR-NLU direction of effect: on SLURP, a gain of up to 5% over the baseline at 15–20% word error rate, and on SLUE entity recognition, a gain of about 2.5% for AR-BINDER II. It must also reproduce, within 0.3 points, the Switchboard result of 48.5% versus 48.4%. No reported result was found on Lacuna for rank correlation between simulated and real recognizer output. Beating a baseline means a higher rank correlation, averaged across target recognizers with bootstrap intervals over test utterances, and a smaller absolute error in estimated method gains.

Why not just…?

Why not just add random typos at matched word error rate? Voice errors often share sound rather than spelling. The "who"/"Hulu" non-prefix case from Tang et al. has little character overlap. Kato's lyric search (Robust and Fast Phonetic String Matching) found that spelling-based matching fails on misheard queries, while matching weighted by acoustic distance succeeds. Still, this alternative is a genuine threat. If understanding models react mainly to error rate, typos could rank methods equally well. The matched-rate comparison settles this directly, and a tie would falsify our claim.

Why not just synthesize audio with text-to-speech and transcribe it? SLURP built 69k synthetic recordings yet still needed real speakers, because synthetic speech lacks spontaneous style and room acoustics. Synthesis also shifts errors toward whatever recognizer the outsider happens to use, not the data owner's recognizer. We include text-to-speech followed by recognition as an extra proxy where compute allows. We expect it to inherit the weaknesses of cross-recognizer substitution.

Why not just use any open recognizer on public audio? This is the strongest alternative, and for generic robustness research it may suffice. It cannot reproduce a specific private system's error profile, which is the whole point when replicating claims like EntSys1M's. Our experiment measures exactly how much fidelity is lost by swapping recognizers versus simulating the right one. If cross-recognizer substitution matches the simulator, we would recommend it and narrow our claim to settings with no matched public audio.

Decisive Checks

The work succeeds if three conditions hold across at least three target recognizers. First, the recognizer-specific simulator achieves a rank correlation of at least 0.8 with real-transcript rankings on both SLURP intents and SLUE entities. Second, it clearly exceeds the typo and uniform-phone baselines. Third, it estimates the gains of robust methods within half their true size, with the largest benefit on PERSON and ORG entities, where AR-NLU reports its largest gains of 7 points over BERT. The noise-on-counts curve then tells data owners how much privacy protection they can add before the proxy stops agreeing with the truth.