Who Is Speaking When the Voice Changes? Conversation Structure as Evidence of Speaker Identity

Speaker diarization is the task of labeling who speaks when in a recording. Current systems do it by clustering short stretches of audio by how the voice sounds. That fails when one person uses many voices: an actor playing a role, or a tabletop role-playing game (TTRPG) player switching between their own voice and a character's. When players in Dungeons & Dragons recordings adopt character voices, the widely used diarizer pyannote.audio goes from a Diarization Error Rate (DER, the fraction of audio time labeled with the wrong speaker, missed, or falsely detected) of 0.12 on office meetings to 0.33. The wespeaker system goes from 0.19 to 0.48 and finds only about 30% of the actual participants (Playing with Voices). The authors trace the failure to missing "lexical awareness": a player who says "I say…" and then growls in an orc voice is obviously still the same person to any listener, but not to an acoustic clusterer.

We propose to let the structure of the conversation vote on speaker identity. Who replies to whom, who asks and who answers, and who frames whose speech all carry evidence about speaker identity that does not change when the voice does. A person rarely answers their own question. A narrator's "she asks" hands the floor to someone the narrator is voicing. Two turns that overlap in time cannot come from the same mouth. We learn a text-based model that predicts whether two utterances come from the same person from these structural cues alone. We train it on text platforms where speaker labels come for free (IRC chat usernames, screenplay character tags), with every platform-specific shortcut removed. We then combine it with acoustic similarity in a single graph over utterances, and infer a two-level identity: many voices, fewer persons.

Core Claim

A conversation graph whose edges come from discourse cues, learned without platform metadata, recovers speaker identity where the voice varies, and this signal transfers across platforms. Concretely, adding a platform-agnostic, text-only "same speaker" edge predictor to acoustic clustering should cut speaker confusion on TTRPG recordings substantially. The gain should concentrate in in-character speech, and it should leave clean meeting audio unharmed.

Two things are new here. Prior graph models of conversation treat speaker identity as a given input, not something to infer. Prior intra-speaker-variability work treats identity as a purely acoustic problem. The cross-platform direction enters as a constraint: the discourse model is only useful if it relies on cues that exist on every platform, so we design and test it for that.

Where the Pieces Come From

Three graph models motivate the view that intra-speaker structure is modelable, but each takes speaker identity as known. DialogueGCN builds an utterance graph whose edges depend on speaker identity, separating a speaker's emotional persistence from others' influence. HC-GNN processes same-speaker and cross-speaker edges in separate branches to avoid mixing their features. The pointer network of Yu & Joty links each chat message to its parent. We invert their input-output relation: the same-speaker edge is what we predict. From HC-GNN we reuse the method of keeping edge types in separate message-passing branches.

The cross-platform literature constrains the design with negative results. When speaker IDs and timestamps are removed, the pointer network's link F1 falls from about 73 to 24 (Yu & Joty). IRC-trained disentanglement fails on screenplays because IRC relies on @-mentions, while drama shifts the conversational floor through movement and pragmatics (Dramatic Conversation Disentanglement). Both results say that a speaker-link model trained naively on chat will learn affordance shortcuts that do not exist in speech. We therefore strip mentions, usernames, timestamps and action lines from all training text. We also train across platforms rather than on one, in the spirit of feature alignment across course platforms in MOOC transfer learning.

The acoustic side supplies both motivation and components. Playing a Part shows that in-domain fine-tuning barely helps on role-played speech, reducing cross-movie EER only from 10.47% to 9.35%, which suggests the acoustic channel alone has a ceiling. Channel adversarial training and local within-conversation PCA (Shum et al.) remove session and channel variation. Neither addresses performed variation, where a single session contains several deliberately different voices from one person. X-vectors meet emotions shows that emotional shifts perturb speaker embeddings, so the problem is not unique to role-play. Sociolinguistic evidence that a speaker's lexical style shifts with their partner (Lexical Entrainment and Intra-Speaker Variability) warns that lexical style alone is also not a stable identity cue. This is why we rely on relational structure (reply, framing, overlap) rather than stylometry.

Research Program

The system has three parts.

Acoustic clustering. An off-the-shelf diarizer segments the audio into utterances and gives acoustic embeddings. We deliberately over-cluster these into "voice" nodes, so that a player's own voice and each character voice can occupy separate clusters.

Transcription and discourse model. A speech recognizer transcribes each utterance. A discourse model then predicts, for pairs of utterances within a window, three things: the probability they share a speaker, a reply-to link, and an explicit framing relation (narrator introducing quoted speech). It is a transformer encoder over utterance text with HC-GNN-style separate branches for reply edges and framing edges.

Joint graph inference. We combine the two channels in a correlation-clustering objective over utterances $i,j$. Correlation clustering partitions a graph by cutting negatively weighted edges and keeping positively weighted ones:

$$\max_{y}\sum_{i<j} y_{ij}\,\big(a_{ij} + \lambda\, d_{ij}\big) \quad \text{s.t. } y \text{ is a valid partition},\; y_{ij}=0 \text{ if } i,j \text{ overlap in time}.$$

Here $y_{ij}\in\{0,1\}$ indicates that $i$ and $j$ are assigned the same person, and $a_{ij}$ is the log-odds of same-speaker from acoustic similarity. The term $d_{ij}$ is the log-odds from the discourse model, and $\lambda$ is a weight tuned on development data. The overlap constraint is a hard cannot-link. Voice nodes are then grouped into person nodes by the resulting partition. Because the number of clusters falls out of the objective, the system never has to guess speaker count up front, which directly targets wespeaker's count failure.

Training data. The discourse model trains on Ubuntu IRC (Kummerfeld et al.), using usernames as same-speaker labels and annotated reply links. It also trains on screenplay dialogue (Chang et al.), using character tags, with action lines removed. Screenplays matter most because they contain quoted and performed speech and floor shifts without @-mentions.

Analysis. Beyond headline numbers, we tag TTRPG segments as in-character or out-of-character and report confusion on each. The claim predicts gains mostly on in-character segments. We also examine learned edge attributions to check whether the model uses framing and reply structure rather than residual shortcuts.

Baselines

The naive reference point is the off-the-shelf diarizers as reported on TTRPG audio. For pyannote.audio this is DER 0.33 on TTRPG versus 0.12 on AMI; for wespeaker it is 0.48 versus 0.19 (Playing with Voices). We rerun both on our splits so that numbers are comparable.

The strongest acoustic competitor is pyannote.audio given everything an acoustic system can reasonably get. That means fine-tuning on the TTRPG training campaigns with the same number of labeled minutes our method sees, and a tuned clustering threshold. We also run a variant given the oracle speaker count, which isolates whether count estimation alone explains the failure. It uses the same audio, the same labeled training time and the same hyperparameter search budget as our system.

The strongest text-aware competitor is a large language model given the diarizer's labeled transcript and asked to relabel speakers. It gets the same development set to choose prompts and the same speech-recognition output. There is no reported result found on Lacuna for LLM speaker correction on TTRPG audio.

Two ablations isolate the key ingredient. The first sets $\lambda=0$, which reduces the system to our acoustic over-cluster plus correlation clustering. The second trains the discourse model with affordance cues left in (usernames, mentions, timestamps, action lines). The second tests the cross-platform claim: it should do better in-domain on IRC but worse when transferred to TTRPG.

Beating the baselines means a statistically significant reduction in speaker-confusion DER on held-out TTRPG campaigns relative to fine-tuned, oracle-count pyannote.audio, and relative to the LLM relabeler. We test significance by bootstrap over sessions. We also require no significant degradation on AMI.

Why not just…?

Why not just fine-tune the diarizer on TTRPG audio? The movie-speech evidence suggests the acoustic channel saturates. In-domain fine-tuning with angular prototypical loss and score normalization only moved cross-movie EER from 10.47% to 9.35% (Playing a Part). With about eight hours of TTRPG audio, data is also scarce. Still, this is a genuine competitor, which is why the fine-tuned diarizer is our main baseline. If it matches our system on in-character segments, the core claim is false.

Why not just give spectral clustering the correct number of speakers? wespeaker's 30% speaker-count recovery makes this tempting, and it may recover much of the gap. The oracle-count baseline measures exactly this. Our prediction is that confusion persists even with the correct count, because the problem is which voices group together, not how many groups there are. If the oracle count closes most of the gap, we would redirect the work toward count estimation from discourse cues.

Why not just have an LLM fix speaker labels from the transcript? This is the most serious threat, since LLMs can read "I say…" and understand reply structure. Its weaknesses are that it sees no acoustic scores, cannot enforce overlap constraints, and has no calibrated way to trade text evidence against voice evidence. It may also inherit transcription errors without recourse. We run it as a full baseline. We also run a hybrid in which the LLM supplies $d_{ij}$ in place of our discourse model. If the hybrid wins, that still supports the central claim that conversation structure should enter a joint graph with acoustics; it would only mean our learned discourse model is not the best source of $d_{ij}$.

Why not just add a rule that merges speech after framing phrases like "I say"? Explicit framing is only one cue, and players often slip into character without it. A rule baseline that merges the next segment with the preceding framer tells us how much of the gain comes from that single cue. The in-character analysis, split by framed versus unframed segments, shows whether reply and floor structure add anything beyond it.

Proposed Experiments

The primary evaluation uses the TTRPG recordings of Playing with Voices, with a leave-one-campaign-out protocol across its six campaigns. If the annotations are not public, we annotate comparable campaigns following their protocol. We report DER decomposed into missed speech, false alarm and speaker confusion, plus speaker-count error. AMI serves as the no-harm control.

A second experiment measures cross-platform transfer of the discourse model on its own. We train on IRC only, screenplays only, and both, then measure same-speaker link accuracy on TTRPG gold transcripts. We also measure link F1 on metadata-stripped Ubuntu IRC, against the reported drop to about 24 link F1 for the pointer network without speaker IDs and timestamps (Yu & Joty).

A third, smaller check applies the same joint inference to movie dialogue. This would pair the screenplay text of Chang et al. with audio of the kind used in VoxMovies. It asks whether gains appear wherever voices are performed, not only in games.

Three outcomes would decide the question. First, whether $\lambda>0$ beats both the fine-tuned oracle-count diarizer and the LLM relabeler on speaker confusion in held-out campaigns. Second, whether that gain sits in in-character, unframed segments. Third, whether the affordance-stripped discourse model transfers better than the affordance-rich one. If all three hold, conversation structure is a platform-portable source of identity evidence that acoustics lack. If the LLM relabeler or the oracle-count baseline matches us, the proposal's mechanism is not needed, and we say so.