Confidence-Gated Phonetic Matching for Few-Shot Spoken Intent Detection

Voice assistants add new commands all the time. The person who defines a new command usually types a handful of example phrases, say five per command. The system then has to recognize that command when users speak it, which means the incoming text comes from a speech recognizer and contains its errors: "turn up the volume" arrives as "turn out the bollom", "Amsterdam" arrives as "Am ster dam". Few-shot intent classifiers, which assign an utterance to a command category from only a few labeled examples, are built and tested on clean typed text. Methods that make language understanding robust to recognition errors assume thousands of paired clean and misrecognized sentences. The deployed case, with clean typed examples, misrecognized spoken queries, and almost no data for the new command, sits between these two literatures, and neither has studied it.

We propose to compare words in two ways when matching a misrecognized query against clean examples: by meaning and by sound. The recognizer's own per-word confidence decides how much each comparison counts. Confidently recognized words are matched by meaning. Words the recognizer was unsure about are matched mainly by how they sound, so "bollom" can still align with "volume". The matching works word by word, using optimal transport (the cheapest way to move one bag of words onto another, given a cost for each word pair). It is trained on episodes built from existing intents, where the examples are clean and the queries are real recognizer output. The claim is that this asymmetric, confidence-gated matching transfers to intents never seen in training. That includes intents whose key words were never misrecognized during training, because sound similarity covers words the model has never seen corrupted.

What is known, and the gap

The few-shot side supplies the matching machinery. DNNC recasts intent detection as asking an entailment model whether a query and a support example mean the same thing. It reaches 91.2% in-domain accuracy at 10 shots on the CLINC150 banking domain. ConvFiT contrastively reshapes a sentence encoder so that nearest-neighbor lookup works, reporting 87.38% 10-shot accuracy on BANKING77. STPN keeps word-level representations and scores a query by an optimal transport cost to the support set. It weights words by query–support lexical overlap, and we reuse that word-level design. All three assume the query and the support come from the same clean distribution. STPN's overlap weighting is exactly what breaks when the query's key word has been misrecognized into a different string.

The recognition-error side motivates the work and also limits the designs available. Can ChatGPT Detect Intent? shows that few-shot prompting reaches 83.93% on SLURP with 20 examples on human transcripts but loses roughly 10–15% on Whisper transcripts. Asking the model to fix the errors does not help, because a text-only model cannot tell that "Siri" and "series" sound alike. That paper documents the drop but proposes no fix. CR-ID adds a phoneme encoder as an "acoustic anchor" and reaches 91.54% on misrecognized SNIPS. PhonemeBERT pretrains jointly on phonemes and transcripts and reports the largest gains at high error rates. Both are fully supervised, and PhonemeBERT notes that phonemes alone lack meaning. That observation argues for mixing the two signals rather than replacing one with the other. ML-LMCL and the consistency-loss approach need large paired clean/noisy corpora to align the two views. Those corpora do not exist for a new intent. Confusion2Vec builds word vectors from recognizer confusion networks, but only for words that appear in its training lattices.

The work closest to ours is in audio, not text. R-ML meta-learns few-shot classifiers over a frozen speech encoder on short command datasets, and it confuses lexically overlapping commands. Generalized zero-shot audio-to-intent builds unseen-intent anchors from synthetic speech. Neither puts clean text examples on one side and recognizer transcripts on the other, and neither uses recognizer confidence when matching.

Core Claim

In the clean-support, noisy-query setting, much of the accuracy loss traces to a small number of low-confidence, intent-bearing words. A word-level matcher can recover a substantial fraction of the loss on novel intents, without any paired data for those intents, if its cost for each word pair shifts from meaning to sound according to the recognizer's confidence in that word.

Research Program

Take a query transcript with words $q_i$ and recognizer confidences $c_i \in [0,1]$, and a clean support example with words $s_j$. The cost of aligning $q_i$ to $s_j$ is

$$C_{ij} = g(c_i)\, d_{\text{sem}}(q_i, s_j) + \big(1-g(c_i)\big)\, d_{\text{phon}}(q_i, s_j).$$

Here $d_{\text{sem}}$ is cosine distance between contextual word embeddings. $d_{\text{phon}}$ is a distance between phoneme sequences, obtained from a pronunciation lexicon with letter-to-sound fallback (as in CR-ID's refinement module) and encoded by a small character-level phoneme network. $g$ is a learned monotone gate. The query-to-class score is the optimal transport cost under STPN-style word weights. We change those weights to use phonetic as well as lexical overlap, so a misrecognized key word is still treated as important.

Transport also handles the "Am ster dam" case: three fragments can each send part of their mass to the single support word "Amsterdam". One-to-one alignment cannot do that. To keep latency close to DNNC-joint's reported 36–41 ms per example, a bi-encoder first retrieves the top candidate intents and transport rescoring runs only on those.

Training is episodic over base intents. Each episode draws clean support transcripts and builds queries from other utterances' real recognizer output, with confidences taken from our own recognizer run. The encoder therefore learns a clean-to-noisy matching function, not a matching function for one distribution. Nothing about the novel intents is seen except their few typed examples.

Baselines

The naive reference is a clean-trained few-shot classifier applied directly to transcripts. We use ConvFiT-style contrastive fine-tuning with nearest-neighbor lookup, and separately DNNC, both trained on base intents and given clean support. Their accuracy on human transcripts versus recognizer transcripts measures the gap we are trying to close. No reported result was found on Lacuna for any few-shot metric method on SLURP recognizer output, so we will produce this number ourselves.

The strongest existing method for this question is few-shot prompting of a large model. The file reports 83.93% (20-shot, oracle transcripts, SLURP) for ChatGPT, with a 10–15% drop on Whisper transcripts (Can ChatGPT Detect Intent?). We will rerun a current instruction-tuned model on our exact novel-intent split, with the same examples and the same transcripts. For it to count as beaten, our method must exceed its recognizer-transcript accuracy at matched shot counts while being far smaller.

As a ceiling, we include the fully supervised pipeline reported in SLURP, with 86.99% action accuracy on perfect transcripts and 81.42% on recognizer output. It is a supervised point of comparison, not a competitor.

Two transfer baselines borrow from the robustness literature. The first is PhonemeBERT used as the few-shot encoder. The second is a CR-ID-style text-plus-phoneme concatenation without gating. Each runs inside the same episodic setup.

The key ablations isolate the gate. We compare fixed mixing weights, pure semantic cost (essentially STPN), pure phonetic cost, and a version with confidences shuffled across words. If shuffled confidences perform as well as real ones, the core claim fails.

Every system gets the same base/novel intent split, the same recognizer outputs, the same support examples, and the same hyperparameter search budget, tuned on held-out base intents only. Beating a baseline means a gain over five or more random split seeds whose confidence interval excludes zero, on recognizer transcripts, with clean-transcript accuracy down by less than one point.

Why not just…?

Why not just run the clean support examples through text-to-speech and then the same recognizer, and add the noisy copies to the support set? This is the most serious threat. It is cheap, needs no training, and produces confusions specific to the novel intent's own vocabulary. It also has known weaknesses. Synthetic speech lacks the disfluencies and far-field noise that drive real errors, a limitation the zero-shot audio-to-intent work shares. And it samples only a few of the possible misrecognitions, whereas our phonetic cost covers the whole neighborhood of similar-sounding words and uses each query's actual confidence. We will run this round-trip augmentation as a baseline and also combine it with our matcher, using SLURP-synth voices to estimate how realistic the synthetic errors are. If round-trip augmentation alone matches us on real SLURP speech, we will report that the problem is mostly a data problem. The gate would then matter only if it adds gains on top of augmentation.

Why not just ask a large language model, possibly after an error-correction step? The file reports that the model's own attempts to correct recognition errors fail because it does not perceive sound similarity (Can ChatGPT Detect Intent?). A separate text corrector needs paired in-domain data that the cold-start setting lacks. It can also overwrite intent words in ways the classifier cannot undo, a risk raised by the consistency-loss work. We include both the prompting baseline and a prompted-correction-then-classify cascade, so the comparison is settled empirically and not by argument.

Why not just feed N-best recognizer hypotheses into an ordinary few-shot classifier? Averaging scores over the top hypotheses is a sensible query-side fix and needs no new matcher. It helps only when the correct word appears somewhere in the list. Our phonetic cost can align a word that appears in none of them. We run N-best score averaging with DNNC as a baseline. We also check whether our gains are concentrated on queries whose correct key word is missing from every hypothesis, which is where the mechanism predicts they should be.

Proposed Experiments

The primary benchmark is SLURP. We split intents into base and novel sets by scenario and transcribe all audio with Whisper, recording word confidences. We evaluate at 1, 5, and 10 shots with clean typed support, reporting accuracy on human versus recognizer transcripts and the fraction of the gap closed: (our accuracy minus naive accuracy) divided by (naive oracle accuracy minus naive recognizer accuracy). Secondary benchmarks are ATIS with recognizer output and SNIPS with the recognizer distortions used by CR-ID, each with intent splits. As a stress test, we use BANKING77 and CLINC150 with synthetic-speech queries, labeled clearly as synthetic.

Three checks decide the claim. First, results are stratified by whether the gold intent's key word was misrecognized. Gains should concentrate there and be near zero on correctly recognized queries. Second, a novel-vocabulary analysis restricts to queries whose misrecognized key word never appeared misrecognized in base-intent training. This tests whether sound similarity, rather than memorized confusions, carries the transfer. Third, the shuffled-confidence and fixed-weight controls must underperform the real gate across error-rate bins. PhonemeBERT's findings suggest the advantage should widen as word error rate rises. If it does not, the confidence signal is not doing the work we claim.