Phonetic Attractors: Why Misheard Questions Produce Confident Wrong Answers

When a question reaches a language model through speech recognition, a voice assistant, or a non-native writer, names are often replaced by words that sound similar but are spelled differently. Sometimes the model quietly recovers the intended name. Other times it answers fluently, and consistently, about a different and more famous entity that happens to sound alike. This note proposes that the second outcome is predictable from the sound-space neighborhood of the intended entity. It also argues that this outcome is mostly invisible to today's hallucination detectors, because those detectors measure whether answers wobble across rephrasings, while this failure does not wobble.

The core claim has two parts. First, hallucination under phonetic noise is governed less by how far the corrupted word is from the correct one than by whether a higher-prior real entity sits nearby in pronunciation, which we call a phonetic attractor. Second, because attractor-driven errors are stable across paraphrases, an uncertainty signal built by perturbing the question in sound space, swapping the entity for its pronunciation neighbors and checking whether the answer changes, should catch them where paraphrase-based sensitivity cannot. If either part is false, the experiments below will show it cleanly.

What the literature establishes and leaves open

Three findings motivate the proposal. Work on LLM transcript repair shows that a model like GPT-4 can reconstruct badly corrupted speech when pronunciation is preserved, even at word error rates above 200% (Aligning ASR Evaluation with Human and LLM Judgments). That paper scores intelligibility for dysarthric speech and does not measure factual answering or hallucination as a function of phonetic distance. Dictionary-based correction of cross-lingual proper nouns, DiP-ASR (Beyond Common Words), finds that GPT-3.5 left alone shows "popularity bias," replacing rare names with frequent English words. Injecting phonetically retrieved candidates fixes much of this. That is the behavior we hypothesize, but DiP-ASR treats it as a nuisance to engineer around rather than as a measurable predictor of downstream hallucination. Finally, the speech hallucination benchmark SHALLOW (Hallucination Benchmark for Speech Foundation Models) separates errors that are acoustically grounded from pure fabrications. It only studies the transcriber, not what a text LLM does with the corrupted transcript.

A fourth line of work constrains the design. Rethinking Hallucinations: Correctness, Consistency, and Prompt Multiplicity splits model behavior into three kinds: consistently correct, consistently wrong ("prompt-agnostic errors"), and answers that flip across prompt variants ("randomness"). It finds that common detectors such as perplexity, entropy, and self-checking mostly detect randomness, not wrongness, and that TruthfulQA errors are mainly of the consistent kind. Similarly, SENSEL (On the Relation between Sensitivity and Accuracy in In-Context Learning) uses disagreement across paraphrased instructions to decide when to abstain, and reports a correlation of up to −0.40 between sensitivity and correctness. Both results imply that any error the model makes consistently will escape sensitivity-based detection. Our hypothesis is that phonetic-attractor errors are exactly such consistent errors, so the gap in the literature is not just "nobody tried homophones." The gap is that the dominant detection approach is structurally blind to this failure.

Adversarial rephrasing work, SECA (SECA: Semantically Equivalent and Coherent Attacks), supplies the cascade mechanism. Once a perturbed prompt pushes the first answer token toward a wrong entity, the model writes a coherent justification. SECA restricts itself to stylistic paraphrase. We add a natural, non-adversarial perturbation family, pronunciation-preserving substitution, that real voice users produce constantly. No paper in the file connects phonetic similarity, prompt sensitivity, and factual hallucination together.

Core Claim, Stated Precisely

For an intended entity $e$ in a question, let $N_\tau(e)$ be the real entities whose phoneme sequences (from grapheme-to-phoneme conversion, or Pinyin for Chinese) lie within normalized phonetic edit distance $\tau$ of $e$. We define the entity's phonetic crowding as

$$ C_\tau(e) = \max_{n \in N_\tau(e)} \big[\log p(n) - \log p(e)\big], $$

where $p(\cdot)$ is a prior over entities estimated two ways: unconditional log-probability of the entity string under the model being tested, and corpus frequency. A positive $C_\tau(e)$ means a more familiar entity sounds like the intended one.

We predict three things. On corrupted questions, the probability that the model answers about the wrong entity rises with $C_\tau(e)$ even after holding fixed the phonetic distance of the corruption itself. Those wrong answers fall disproportionately into the "prompt-agnostic error" category of the multiplicity taxonomy. And a phonetic neighborhood sensitivity score separates these errors from correct answers better than paraphrase sensitivity. That score is built by re-asking the question with the entity replaced by a few members of $N_\tau(e)$ and the intended form, then measuring how often the answer's referent shifts.

Research Program

The first component is a controlled corruption benchmark built from existing factual QA. For each entity-bearing question, we generate corrupted variants at several phonetic distances. Half of the variants push toward a real neighbor, so the attractor is present. The other half push toward a non-word at matched phonetic and character distance, so no attractor is present. Corruptions are produced by G2P-space edits mapped back to plausible spellings, following the dictionary-matching recipe of DiP-ASR. We also collect natural corruptions by running a speech recognizer on the spoken side of CCFQA (CCFQA), which has parallel speech and text questions in eight languages. The natural set guards against our synthetic edits being unrealistic.

The second component is the analysis. We fit hallucination probability as a function of corruption distance, $C_\tau(e)$, and their interaction, per model. Each instance is labeled with the multiplicity taxonomy by also asking several paraphrases of the corrupted question. Following SECA, we separately check whether the first content token already names the attractor. If it does, the cascade starts at the entity-resolution step rather than later in the reasoning.

The third component is the detector. Phonetic neighborhood sensitivity uses $k$ extra queries per question, with neighbors chosen from $N_\tau(e)$. It is used for selective prediction, meaning the model answers only when confident and abstains otherwise. Its key design choice, perturbing along the axis where the error lives, is isolated by ablations described below.

Baselines

The naive reference point is to answer the corrupted question as given, with no correction or abstention. Comparing it with the clean question gives the raw cost of phonetic noise. The CCFQA paper already reports one form of this cost: GPT-4o-mini keeps only 62.7% cross-modal consistency between speech and text versions of the same question, and its audio variant reaches 40.4% accuracy on spoken tasks. We will reproduce the text and transcript conditions on our models before building anything on top.

The strongest correction competitor is normalize-then-answer, in two forms. One is DiP-ASR style: phonetic and spelling dictionary retrieval of candidates followed by LLM selection. DiP-ASR reports reducing complex Hindi word error rate from 45.46% to 29.94%, with N-best lists alone giving about 3.5%. The other is plain LLM self-correction with the instruction that the question "may contain speech recognition errors." Both get the same models, the same entity dictionary, and the same number of model calls as our detector. Beating them means higher factual accuracy at full coverage, or, if the attractor hypothesis is right, showing that their residual errors are concentrated at high $C_\tau$ and are caught by our score.

The strongest detection competitor is SENSEL-style paraphrase sensitivity. SENSEL reports a +4.1 point gain in area under the F1–coverage curve over probability-based methods on GPT-J-6B, a curve that tracks accuracy as the model is allowed to answer more questions. We run it with exactly $k$ paraphrases, matching our $k$ phonetic probes. We also run self-consistency sampling and single-pass maximum token probability and entropy at matched budget. We report no reported result found on Lacuna for selective prediction on CCFQA, TruthfulQA, or Med-HALT under phonetic noise, so all detection numbers there are new and are compared only within this study. Beating SENSEL means a higher area under the accuracy–coverage curve overall, and specifically higher recall of wrong answers on the high-crowding subset.

Two ablations isolate the key ingredient. The first replaces phonetic neighbors with orthographic neighbors at matched character edit distance. If this does as well, sound space is irrelevant and the claim reduces to typo robustness. The second restricts phonetic probes to non-word neighbors. If this does as well, the attractor structure is irrelevant and any local perturbation suffices.

Why not just…?

Why not just correct the spelling first and then answer? This is the most serious threat, and DiP-ASR shows dictionary-guided correction works well on proper nouns. But the corrector is itself a language model with the same popularity bias DiP-ASR documents. We expect it to "fix" a corrupted rare name to its famous neighbor, which turns the error into a confident one. The experiment settles this directly. If normalize-then-answer leaves no residual error that grows with $C_\tau$, the detector is unnecessary and we report that the attractor problem is solvable by preprocessing.

Why not just use paraphrase sensitivity or self-consistency, which already work? The multiplicity paper finds that such detectors track randomness rather than correctness, and that consistent errors evade them. If attractor errors are consistent, paraphrasing a question about the wrong entity keeps yielding the same wrong entity. Our stratified comparison on high-crowding instances tests exactly this. If SENSEL matches our score there, the hypothesis that these errors are prompt-agnostic is false.

Why not just perturb with typos, which is cheaper? The orthographic ablation answers this. The motivating case for sound space is that a misheard word can look very different while sounding identical, as with "beach" transcribed as "bitch" in children's content (Beach to Bitch) or "jhumka" heard as "dumka" in DiP-ASR. Character-close neighbors therefore often miss the true attractor.

Why not just use a bigger model? TruthfulQA (TruthfulQA) shows larger models can be less truthful because they absorb popular content more strongly. That is the same pressure that would strengthen attractors. We test at least two sizes per family and report whether the crowding effect shrinks or grows with scale.

Proposed Experiments

The datasets are CCFQA for natural speech corruptions across languages, including Pinyin-based neighborhoods for Mandarin. We use Wiki-FACTOR and TruthfulQA questions with entities for synthetic corruption, and Med-HALT for the high-stakes case of drug and condition names, where sound-alike substitutions are clinically dangerous. On Med-HALT, the multiplicity paper already reports about 60% ambiguity for Llama2-13B-Chat. Models include open families at two sizes, with access to output probabilities, plus GPT-4o-mini through its API. Answers are scored by exact match on multiple-choice formats and by an NLI-based judge for free text. Each wrong answer is labeled as attractor-referent or other.

The decisive checks are three. First, in a regression of wrong-entity rate on corruption distance and crowding, crowding must carry a significant positive coefficient with distance held fixed. Second, the attractor-present condition must produce a higher share of prompt-agnostic errors than the matched non-word condition. Third, phonetic neighborhood sensitivity must beat paraphrase sensitivity on area under the accuracy–coverage curve at equal query budget, and must beat both ablations. A null on the first check falsifies the attractor idea. A null on the second removes the reason paraphrase detectors should fail. A null on the third means the diagnosis is right but the cheap probe is not the remedy, which would still be worth reporting.