Nearest-neighbor voice conversion makes one person's speech sound like another's in a simple way. Each short slice of the source speech is replaced by the most similar slices recorded from the target speaker, and the result is turned back into audio. This works well when the target recording contains every sound the source needs. With only three seconds of target audio, many sounds are simply missing. The matcher then grabs the closest available slice, which is often the wrong sound in the right voice, and intelligibility collapses. The fix the field has adopted is to invent more target material: a generator "hallucinates" thousands of extra target-voice slices so that something close to every sound exists in the pool.
We argue that the failure is mainly a ranking problem, and that generating more material treats the symptom. The matcher sorts candidate slices by one distance that mixes up what sound a slice is with whose voice it is. When the right sound is absent, that single number cannot tell "wrong sound, right speaker" apart from "right sound, slightly different acoustics." We propose to keep the matcher as a single-objective selector, but with a different total order. Candidates are ranked first by whether they carry the source's phone (the speech sound), then by whether they were really recorded rather than hallucinated, and only then by acoustic distance. Hallucination is invoked only for the phones the source needs and the real reference lacks. The prediction is that this recovers the intelligibility gains of full hallucination while putting far fewer invented slices into the output, and therefore keeps more of the target speaker's identity.
Core Claim
The setting is k-nearest-neighbor voice conversion (kNN-VC). It represents speech as frames of WavLM features, which are vectors from a self-supervised speech model trained without labels. Each source frame is replaced by the average of its $k$ nearest target frames. Phoneme Hallucinator documents the coverage gap. At a 3-second target reference, plain kNN-VC reaches 45.92% word error rate (WER) on LibriSpeech. Expanding the target pool with a conditional variational autoencoder and Set Transformer brings this to 5.10%. That paper's generator is trained with a single likelihood-style objective and expands the pool without regard to which sounds the source actually needs. Its authors also note that vocoding hallucinated features degrades quality. NeuCoSVC supplies the constraint on our design. It shows that frames foreign to the target pool cause artifacts and timbre leakage, and it eliminates leakage by restricting synthesis to genuine target frames, at the price of failing when sounds are absent. Taken together, these two papers mark out a trade-off between intelligibility and voice identity. Hallucinate everything and you gain intelligibility while diluting identity. Hallucinate nothing and you keep identity while losing intelligibility.
Our claim is that most of this trade-off is an artifact of the selection rule rather than a true Pareto conflict. The proposed rule is a lexicographic ranking, meaning one compares criteria strictly in order and uses the next only to break ties. For a source frame $x$ with predicted phone $p(x)$, each pool frame $y$ has a phone label $c(y)$, an origin flag $h(y)=1$ if hallucinated and 0 if real, and WavLM cosine distance $d(x,y)$. The $k$ selected neighbors minimize, in lexicographic order,
$$ \big(\,\mathbb{1}[c(y)\neq p(x)],\;\; h(y),\;\; d(x,y)\,\big). $$
This is still a strict total order and therefore a single objective. It needs no weights and no Pareto front. A real frame of the correct phone always beats a hallucinated one. A hallucinated frame of the correct phone always beats a real frame of the wrong phone. Hallucinated frames therefore enter the output only where the real reference has a true coverage hole.
The optimization literature informs this choice in two ways. Zhang et al. show that when objectives genuinely conflict, Chebyshev scalarization recovers Pareto points that weighted sums miss, and their WC-Penalty method reaches hypervolume 2.47e-2 against MoCo's 1.66e-5 (A Tale of Two Problems). That route requires stating the conflict as competing objectives and carefully tuning penalty schedules. We take the opposite bet: here the objectives have a natural priority, so a total order suffices. The evidence that this bet can work comes from the same literature. USEA relies on scalar sortability to decide which candidates are worth expensive evaluation, and its authors name the loss of a total order as the obstacle to extending it. A lexicographic order keeps that sortability. If our bet is wrong, the experiments will show a real frontier that the ordering cannot reach, and we report that outcome rather than hide it.
Research Program
We will build three components on top of the public kNN-VC and Phoneme Hallucinator pipelines, keeping their WavLM layer and vocoder fixed.
The first component is a frame-level phone classifier on WavLM features. Following the phone predictor that Private kNN-VC trains on LibriSpeech train-clean-100, we train it with forced-alignment labels. It labels source frames and real target frames. To limit damage from its errors, we use the predicted phone only when posterior confidence exceeds a threshold tuned on dev-clean. Otherwise the first ranking criterion is dropped for that frame and matching falls back to distance.
The second component is coverage-targeted hallucination. For each conversion we list the phones present in the source utterance that have fewer than $m$ confidently labeled frames in the real reference. We then draw from the frozen Phoneme Hallucinator generator, keeping only samples the classifier assigns to those missing phones, until each has $m$ frames. This is rejection sampling, and it is affordable because the generator produces 10 minutes of features in 1.48 seconds on an RTX 4090 (Phoneme Hallucinator).
The third component is the lexicographic matcher above, followed by the unchanged vocoder.
The analysis half of the program tests whether the diagnosis is right. We define a phone substitution rate: the fraction of source frames whose selected neighbors carry a different phone label than the source frame. We measure it as a function of reference length for plain kNN-VC, for full hallucination, and for our method. If the coverage-gap story is correct, this rate should track WER across conditions. If WER at short references is instead dominated by vocoder or prosody failures, the substitution rate will not explain it, and the proposal's premise is wrong.
Baselines
The do-nothing reference is plain kNN-VC with the same 3-second reference, same WavLM layer, same $k$ and same vocoder. Its reported 3-second WER on LibriSpeech is 45.92% (Phoneme Hallucinator).
The strongest existing method is Phoneme Hallucinator with full unguided pool expansion. It reports 5.10% WER at 3 seconds on LibriSpeech in the same paper. We found no reported speaker-similarity number for it on Lacuna, so we will measure speaker similarity for all systems ourselves. It is the strongest competitor because it targets exactly the coverage gap under exactly our data regime. We run the authors' released generator unchanged and tune its expansion size on dev-clean with the same tuning budget we give our own thresholds. Beating it means WER no worse than its measured WER plus a margin fixed in advance on dev-clean, together with a paired, statistically significant gain in speaker similarity to the target on test-clean.
The tuned weighted-sum matcher replaces the lexicographic order with cosine distance plus $\lambda$ times a phone-mismatch penalty plus $\mu$ times a hallucination penalty. $\lambda$ and $\mu$ are swept on a grid with the same budget, in the style of the grid sweeps of Private kNN-VC. This is the baseline that isolates whether strict ordering matters or whether "add a phone term" is the whole story.
FreeVC is a parametric reference that does no retrieval at all. It uses spectrogram-resize augmentation and an information bottleneck in a VITS model (FreeVC), and reports MOS 3.99 against YourTTS's 3.46 on VCTK seen-to-seen. We run its released model on our LibriSpeech references.
Four ablations isolate the key ingredients:
- Lexicographic matching over the real pool only, with no hallucination.
- Targeted hallucination with plain distance matching.
- Full ranking without the real-over-hallucinated criterion.
- Oracle phone labels from forced alignment, which bounds how much classifier error costs.
Why not just…?
Why not just tune how much Phoneme Hallucinator generates? Shrinking the hallucinated pool trades intelligibility for identity along one knob. Every source frame faces the same pool, so phones the reference already covers still compete with invented frames. Our pool-size sweep traces this one-dimensional curve, and the proposal succeeds only if our operating point lies outside it. If the tuned curve reaches the same point, targeting adds nothing and we will say so.
Why not just add a phone-mismatch penalty to the distance and tune the weight? This is the real threat, because the lexicographic order is the limit of that weighted sum as the penalty grows. The swept weighted-sum baseline settles the question. If some finite weight matches or beats strict ordering, the contribution reduces to phone-aware matching with one tuned weight, which is simpler, and we would adopt it. The case for ordering rests on its needing no per-reference-length tuning. We test that by applying dev-tuned weights across reference lengths from 3 to 60 seconds and checking whether their best value drifts.
Why not just use multi-objective optimization over the conversion settings? The knobs here are low-dimensional, few enough that a grid sweep covers them. Standard Gaussian-process Bayesian optimization was about 1,000 times slower than a population-based method on 50-dimensional problems (USEA). Pareto methods such as WC-Penalty bring penalty-scaling sensitivity (A Tale of Two Problems). Our weighted-sum grid already traces the empirical frontier. If it reveals non-convex regions the ordering misses, that is evidence for the multi-objective route.
Why not just use a parametric converter that never retrieves frames? FreeVC conditions on a single global speaker vector, so it has no explicit inventory problem. That same vector carries little of the target's fine-grained phonetic detail. The comparison on identical 3-second references shows directly which family handles impoverished references better.
Proposed Experiments
The main experiment uses LibriSpeech test-clean sources, following the Phoneme Hallucinator protocol. Target references are cut to 3, 5, 10, 30 and 60 seconds, the last chosen because NeuCoSVC observes plateaus near 60 seconds. All tuning happens on dev-clean. We measure four things:
- WER from a fixed ASR model.
- Speaker similarity to the target, as cosine similarity from a fixed speaker-verification model and as equal error rate against held-out target audio.
- The fraction of output frames drawn from hallucinated material.
- A listening test for naturalness and similarity on a subset.
LibriTTS train-other-500 speakers, as used in Private kNN-VC, provide a second set of target voices to check that results are not specific to particular speakers.
Three checks are decisive. The first is the diagnosis: the phone substitution rate should rise as references shorten, should explain most of plain kNN-VC's WER, and should be driven near zero by our matcher. The second is the main comparison against tuned full hallucination. The prediction is equal WER with fewer hallucinated frames in the output and higher speaker similarity. If similarity does not improve, the identity-dilution argument inherited from NeuCoSVC does not transfer to speech and the method has no advantage. The third is the comparison against the tuned weighted sum across reference lengths, which settles whether a strict total order is needed.
A secondary stress test applies the same matcher to cross-domain conversion with speech references and singing sources on OpenSinger and NUS-48E. NeuCoSVC reports that this setting degrades sharply, against its in-domain similarity of 0.57. Its hard target-only constraint is exactly what our real-before-hallucinated tie-break relaxes, and only where a phone is missing.
References
- Phoneme Hallucinator: One-Shot Voice Conversion via Set Expansion
- Neural Concatenative Singing Voice Conversion: Rethinking Concatenation-Based Approach for One-Shot Singing Voice Conversion
- A Tale of Two Problems: Multi-Task Bilevel Learning Meets Equality Constrained Multi-Objective Optimization
- Un-evaluated solutions may be valuable in expensive optimization
- Private kNN-VC: Interpretable Anonymization of Converted Speech
- FreeVC: Towards High-Quality Text-Free One-Shot Voice Conversion