New speech systems built on large language models identify speakers by reading a list of enrolled people in their prompt. Each entry pairs a name with a short reference recording, and the model then writes a transcript tagged with who said what. This design is convenient, and it beats older multi-stage pipelines on meeting transcription. It also inherits a well-documented weakness of language models: their answers can change when the order or wording of a prompt changes in ways that should not matter. For speaker identification that weakness is a security and privacy problem. The case that matters most in real deployments is a guest, someone not on the list. The system should say "unknown." Nobody has checked whether it does, or whether it confidently gives the guest the name of whichever enrolled person happens to sit in a favourable position in the prompt.
This note proposes to measure that failure and then use it. The test is simple. Shuffle the order of the enrolled list and swap the names for arbitrary placeholders. Neither change alters which voice belongs to whom. If the model's attribution of a speech segment survives these changes, the segment is anchored in the audio. If the attribution drifts, the model is guessing, and in an open-set setting a guess usually means a guest. We ask three questions. Is attribution drift under these changes a better guest detector than the model's own token probabilities? Does it add information beyond plain acoustic similarity? Does it remain useful after the model has been fine-tuned to resist it?
Core Claim
Two kinds of identification are in play here. Open-set speaker identification means naming enrolled speakers while rejecting speakers who were never enrolled. Prompt-registered speech LLMs are speech-to-text models that receive the enrolled speakers inside their prompt. We claim that open-set identification in these models fails in a specific way: when the model names an unenrolled speaker, that name depends on where each candidate appears in the prompt and what label it carries. We further claim that this dependence, measured with perturbations that provably preserve the correct answer, is a stronger rejection signal than single-prompt confidence. On acoustically confusable galleries, it is also complementary to acoustic embedding similarity.
The work closest to this is SpeakerLM. It couples a SenseVoice-large audio encoder to Qwen2.5-7B-Instruct and puts enrolled speakers in the prompt. Its hardest setting, "Over-Regist," lists up to 50 candidates of whom only some actually speak. Every voice in the audio is still on the list, though. The paper tests neither unenrolled guests nor sensitivity to gallery order or naming. The perturbation-as-uncertainty idea comes from SENSEL, which abstains on text classification when outputs vary across instruction paraphrases and demonstration orders. It finds a correlation of up to −0.40 between sensitivity and accuracy, and a gain of 4.1 points in area under the F1–coverage curve over MaxProb on GPT-J-6B.
Our proposal differs from SENSEL in two ways. First, the domain: acoustic open-set biometrics rather than text classification. Second, and more important, the type of perturbation. A paraphrased instruction can subtly change what is being asked. A permuted, relabeled gallery cannot change the correct identity, so any change in the output is an error of the model and not an ambiguity of the task.
Research Program
The first component is a measurement study. Using SpeakerLM's architecture, we build open-set test conditions in which the prompt gallery omits one or more people who actually speak (the guests). The gallery is padded with non-speaking distractors, as in Over-Regist. We then apply three kinds of perturbation. The first permutes gallery order. The second renames candidates with real names, arbitrary placeholder names, or symbols, following the label-swap manipulations that moved text leaderboards by up to eight positions in When Benchmarks are Targets. The third reformats the registration template.
We report several quantities. One is how often guests receive an enrolled name. Another is whether these false acceptances concentrate on early or late gallery positions, the primacy and recency effects described in The Order Effect. A third is the prompt-multiplicity "ambiguity" of Rethinking Hallucinations: the share of segments whose identity changes across perturbations, reported separately for enrolled speakers and guests.
The second component is the rejection score. For a decoded segment $s$, we run $K$ perturbed prompts. Each applies a random reordering and a random renaming map $\sigma_k$, which records which displayed label stands for which enrolled identity. We translate each output label $y_k$ back to its underlying identity using that map. The consistency score is the largest fraction of runs that agree on any single identity $i$:
$$c(s) = \max_i \frac{1}{K}\sum_{k=1}^{K} \mathbf{1}\left[\sigma_k^{-1}(y_k) = i\right]$$
The model accepts the majority identity when $c(s) \ge \tau$ and outputs "unknown" otherwise. The threshold $\tau$ is set on held-out data to hit a target false-alarm rate. Segments are matched across runs by aligning transcript tokens, in the same spirit as the concatenated minimum-permutation error rate. In that alignment, a segment that one run attributes to "unknown" or omits counts as disagreement. We also train a small logistic fusion of $c(s)$ with an acoustic cosine score. This fusion tests complementarity directly.
The third component is a training fix. During fine-tuning we randomly permute and relabel the gallery, and we include guest speakers with an explicit "unknown" target. This is the natural next step after observing brittleness. It also tests whether our score remains useful once training has been aimed at the brittleness.
A secondary stress test follows from the tension between CASA-ASR, which shows that linguistic context reduces attribution errors, and ProSA, which ties prompt sensitivity to flat decoding confidence. We insert guest utterances whose content names an enrolled person, as in "this is Wang speaking," and measure whether text cues override the voice.
Baselines
The do-nothing reference is SpeakerLM-style decoding with a single fixed prompt in canonical gallery order and no "unknown" option. It measures how much the existing design hallucinates enrolled names onto guests. For this setting we found no reported result on Lacuna. The only reported SpeakerLM number is closed-set: 16.05% cpCER on AliMeeting-Eval, against 23.20% for the best cascaded system (SpeakerLM). We will first reproduce a number close to 16.05% in the Over-Regist setting. This confirms that our reimplementation, or a released checkpoint if one is available, is faithful before we move to open-set conditions.
The prompt-only fix adds "unknown" as an allowed label and instructs the model to use it. It costs nothing, and it shows whether instruction alone recovers rejection.
The single-prompt confidence baseline thresholds the probability of the generated name tokens, either the maximum or one minus the entropy. This is the alternative that SENSEL beat on text. Beating it here means a higher detection-and-identification rate at fixed false-alarm rates, using the same model and the same held-out threshold calibration.
The strongest competitor for guest rejection is purely acoustic. Each decoded segment is embedded with a speaker encoder, Half-ResNet34 as in OpenFEAT, and scored by cosine similarity against each enrolled profile. We also include an OpenFEAT-adapted variant, which reported 23% to 31% relative reduction in identification equal error rate on hard VoxCeleb households. We tune the acoustic threshold on the same calibration split and with the same tuning budget as our score.
Two ablations isolate the key ingredient. The first is compute-matched resampling: $K$ stochastic decodes of the unperturbed prompt at nonzero temperature. It tests whether the signal comes from label-preserving perturbation or simply from drawing several samples. The second is paraphrase-only SENSEL: $K$ instruction paraphrases with fixed gallery order and names. It tests whether gallery permutation and renaming carry more information than wording changes do.
Our method beats a baseline if it raises the detection-and-identification rate at false-alarm rates of 1% and 5%, with confidence intervals bootstrapped over sessions. The bar against the acoustic baseline is stricter. Fusing $c(s)$ with cosine must significantly improve on cosine alone. Matching cosine is not enough to count as a win.
Why not just…?
Why not just verify the LLM's chosen name with an acoustic embedding and a threshold? This is the most serious threat. We treat it as a baseline we must add to, not one we must beat alone. Universal background models, distance thresholds such as the Earth Mover's Distance method of CCC SRE 2006, and cosine scoring all handle easy guests well. Two documented conditions make acoustic scoring fail. OpenFEAT shows universal embeddings break down when speakers are acoustically similar. VoxBlink2 reports that open-set identification degrades as the gallery grows. Short, overlapped meeting segments make both conditions worse. The fusion experiment, stratified by acoustic similarity and gallery size, settles the question. If $c(s)$ adds nothing even on the hardest stratum, we will report that the consistency score is redundant and that the measurement study is the contribution.
Why not just threshold the model's own probability on the name token? Evidence from text models argues against this. SENSEL found that MaxProb and entropy are distorted by label bias and stay confident on wrong answers. ProSA links flipped outputs to flat decoding distributions. A guest's name assignment could still be confidently wrong in a position-driven way, and that is precisely what single-prompt confidence cannot see. The head-to-head comparison on identical data decides this.
Why not just fine-tune with guests and randomized galleries so the brittleness disappears? This could plausibly work, so we build it as the third component instead of arguing against it. If permutation-and-renaming training with explicit "unknown" targets drives guest false acceptance to the acoustic baseline's level, and also flattens the consistency signal, the practical recommendation becomes "train this way." That outcome is still informative, because it confirms brittleness was the cause. We expect partial invariance on training-like galleries and residual drift on larger or unfamiliarly formatted ones. We test this by evaluating the fine-tuned model at gallery sizes and template styles held out from training.
Proposed Experiments
The main testbeds are AliMeeting-Eval and the AISHELL-4 evaluation set, which has 20 sessions with 4 to 8 speakers and about 19% overlap (AISHELL-4). For each session we register all participants except one or two held-out guests. We add distractors drawn from other sessions to reach gallery sizes of 5, 20, and 50, matching SpeakerLM's Over-Regist maximum. We test in-car generalization on AISHELL-5 (AISHELL-5), for which the file reports no SpeakerLM number.
A second testbed covers the acoustically hard regime. We build synthetic multi-turn conversations from VoxCeleb households selected, as in OpenFEAT, from the top 15% of cosine similarity, and add guests drawn from VoxBlink2 speakers. The metrics follow VoxBlink2's protocol: detection-and-identification rate at fixed false-alarm rates. We also report guest false-acceptance rate, cpCER for transcription quality, and multiplicity ambiguity.
The decisive checks are three. First, guest false acceptance must vary significantly with gallery position or label under the do-nothing model; otherwise the brittleness premise fails. Second, at matched $K$, $c(s)$ must beat both temperature resampling and paraphrase-only perturbation, which would show that answer-preserving gallery perturbation is the active ingredient. Third, fusing $c(s)$ with acoustic cosine must improve the detection-and-identification rate on the high-similarity and large-gallery strata, both before and after invariance fine-tuning. We will sweep $K$ from 2 to 16 to report the trade-off between rejection quality and inference cost.
References
- SpeakerLM: End-to-End Versatile Speaker Diarization and Recognition with Multimodal Large Language Models
- On the Relation between Sensitivity and Accuracy in In-Context Learning
- When Benchmarks are Targets: Revealing the Sensitivity of Large Language Model Leaderboards
- The Order Effect: Investigating Prompt Sensitivity in Closed-Source LLMs
- Rethinking Hallucinations: Correctness, Consistency, and Prompt Multiplicity
- CASA-ASR: Context-Aware Speaker-Attributed ASR
- ProSA: Assessing and Understanding the Prompt Sensitivity of LLMs
- OpenFEAT: Improving Speaker Identification by Open-Set Few-Shot Embedding Adaptation with Transformer
- Speaker Identification Method Using Earth Mover's Distance for CCC Speaker Recognition Evaluation 2006
- VoxBlink2: A 100K+ Speaker Recognition Corpus and the Open-Set Speaker-Identification Benchmark
- AISHELL-4: An Open Source Dataset for Speech Enhancement, Separation, Recognition and Speaker Diarization in Conference Scenario
- AISHELL-5: The First Open-Source In-Car Multi-Channel Multi-Speaker Speech Dataset for Automatic Speech Diarization and Recognition