Speech language models are large models that take audio in and answer questions about it. When two or more people talk, they often answer "who said what" questions by guessing from the words rather than listening to the voices. This proposal tests a geometric account of why, and a cheap fix. Self-supervised speech models are networks pretrained on unlabeled audio. Prior work found that these models store speaker identity in a set of directions roughly perpendicular to the directions that store phonetic content. If that holds, then within the speaker directions a single talker should trace one stable direction over a short window. A change of turn should look like an abrupt rotation of that direction. Two people talking at once should look like a blend of two directions. Radio astronomers face the same three-way distinction when they read pulsar and fast-radio-burst light, which arrives as two perpendicular polarization modes. A clean flip between modes shows up as a sudden 90° jump in polarization angle. Two modes present at once show up as a drop in how polarized the light is.
The proposal borrows that diagnostic. From the speaker directions of a pretrained encoder, it computes a per-frame "speaker polarization degree" and a "rotation angle." It uses them in two ways. The first is a training-free probe of whether speaker geometry survives overlapping speech. The second is a small set of extra input tokens that let a speech language model answer attribution questions directly, without first writing out a speaker-tagged transcript. If the geometry holds, the result is a low-latency attribution signal grounded in the voices, aimed at the failure that current models show under overlap.
Core Claim
The claim is that overlap and turn-taking are separable in the speaker subspace of a pretrained speech encoder, using only second-order statistics. Let $h_t$ be a frame representation from a chosen encoder layer. Let $P$ project onto a $k$-dimensional speaker subspace found by principal component analysis on speaker-labelled single-talker data. Let $C_t^{(w)}$ be the covariance of the projected frames $P h_\tau$ over a window of $w$ frames centred at $t$, with top two eigenvalues $\lambda_1 \ge \lambda_2$ and leading eigenvector $u_t^{(w)}$. We define
$$ p_t^{(w)} = \frac{\lambda_1 - \lambda_2}{\lambda_1 + \lambda_2}, \qquad \theta_t^{(w)} = \arccos \left| u_t^{(w)\top} u_{t-w}^{(w)} \right| . $$
Here $p$ is the analogue of the degree of linear polarization. It is near 1 when one voice dominates the window and falls toward 0 when two voices contribute equally. $\theta$ is the analogue of the polarization-angle jump between neighbouring windows.
A window that straddles a clean turn change will also look depolarized, because two directions are averaged over time. This is exactly the "superposition versus discontinuous switching" ambiguity left open for FRB 20201124A in Sudden Polarization Angle Jumps of the Repeating Fast Radio Burst FRB 20201124A. That paper reproduced jumps accompanied by polarization drops by incoherently summing two orthogonally polarized components.
Our resolution is a scale test. For a true switch, shrinking $w$ restores $p$ toward 1 on both sides and concentrates $\theta$ into a spike. For true overlap, $p$ stays low at every window size down to the encoder's frame rate. The new part is the multi-scale signature $\{p^{(w)}, \theta^{(w)}\}$. It is never used to erase speakers, only to locate and track them, and it then serves as a compact grounding signal for attribution reasoning.
Where This Sits
Self-supervised Predictive Coding Models Encode Speaker and Phonetic Information in Orthogonal Subspaces supplies the method. It shows that APC and CPC representations of clean speech place phonetic and speaker variation in nearly perpendicular subspaces, and it projects the speaker subspace away to help phone discrimination. It never studies overlap, never runs the inverse operation of tracking speakers, and never examines masked-prediction encoders.
WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing constrains the design. It pretrains with simulated overlapping talkers but asks the model to reconstruct only the primary speaker. Its upper layers may therefore suppress the second voice and look falsely polarized. WavLM also reports that speaker information lives in lower and middle layers. We therefore treat the choice of layer as an experimental variable rather than an assumption.
The polarization analogy and the incoherent two-component model come from Sudden Polarization Angle Jumps of the Repeating Fast Radio Burst FRB 20201124A and A New Emission Mode of PSR B1859+07. The latter used clean orthogonal jumps to argue that apparent timing shifts are mode switches inside a fixed emitter. We borrow the diagnostic logic and claim no physical correspondence.
The motivating evidence is HEAR Who Said What: Unlocking Speaker-Attributed Reasoning via Counterfactual Voice Grounding. It builds the HEAR benchmark and the CASH dataset, in which transcripts are held fixed while voices are swapped. It shows that speech language models guess attribution from text and collapse under overlap. Its A2R model fixes much of this with reinforcement learning, by rewarding the model for writing a full speaker-tagged transcript before reasoning. The authors themselves flag the latency of that step and their reliance on synthetic audio.
Two strands of prior work bring speaker information into the model in different ways from ours. Speaker-attributed ASR systems such as End-to-End Speaker-Attributed ASR with Transformer and On Speaker Attribution with SURT solve transcription plus attribution with dedicated decoders. The first requires pre-enrolled speaker profiles. The second tracks speakers by order of appearance and degrades at four overlapping talkers. Text-side work such as Evaluating the Effects of Embedding with Speaker Identity Information in Dialogue Summarization shows that speaker information helps only when placed in low-interference dimensions. That is a text-domain echo of the claim that speaker signals should live in their own subspace.
Research Program
The work proceeds in three stages, each gated on the previous one.
The first stage is geometric analysis. We extract speaker subspaces layer by layer from WavLM and HuBERT-style encoders on single-talker LibriSpeech. We test whether the phone/speaker orthogonality reported for APC and CPC holds for these encoders. We then measure $p$ and $\theta$ on LibriSpeechMix-style mixtures in which overlap ratio and relative loudness are controlled.
The predictions are falsifiable. In the encoder's speaker-preserving layers, $p$ should fall roughly as the two-component model predicts when relative loudness approaches equality, the way linear polarization drops when two modes have equal power. In upper layers, $p$ should stay high because the second voice is suppressed. If no layer shows depolarization under overlap, the speaker subspace does not superpose, and the project pivots to reporting that negative result.
The second stage is a training-free detector. Thresholds on the multi-scale signature yield frame labels: single talker, turn change, or overlap. We score these as overlap detection and speaker-change detection on LibriCSS, AMI and CALLHOME, without fine-tuning.
The third stage is attribution tokens for a speech language model. At a fixed stride, we pool the projected speaker coordinates $P h_t$ together with $p_t$ and $\theta_t$ into a few "mode tokens." A small adapter appends these to the audio tokens of an open speech language model. The tokens are low-dimensional by construction, so they add little sequence length. We train on CASH with HEAR's counterfactual hard negatives, under the same reinforcement-learning recipe as A2R, but reward the final answer directly with no transcription step. The model can then match voices across turns by comparing directions in the speaker subspace, and can recognise overlapped spans from low $p$, instead of inferring both from the words.
Baselines
The naive reference point is the untrained base speech language model evaluated zero-shot on HEAR, with no extra tokens. On HEAR's counterfactual voice-swap pairs, the open-source models reported in HEAR Who Said What reach 14.9% paired accuracy, close to chance. Any method must clear this floor by a wide margin on the same pairs.
The strongest existing method is A2R from the same paper. It reports an average HEAR score of 67.1% and paired accuracy of 60.5% at 30B parameters, against 88.8% average for Gemini 3.1 Pro. Its scale and transcription-first reward make the published number a moving target, so we also re-run the A2R recipe on our own base model. That re-run uses the same CASH data, the same GRPO budget measured in update steps and rollouts, and the same hyperparameter search budget.
Beating A2R means two things. On the overlap subset of HEAR, which the paper describes only as collapsing with no exact number reported on Lacuna, we must match or exceed its paired accuracy. On all of HEAR, we must stay within a few points of its paired accuracy while cutting time-to-first-answer-token substantially. No reported latency figure for A2R was found on Lacuna, so we measure both systems on identical hardware.
The diarization-token baseline is the most dangerous competitor. We run an end-to-end neural diarization model of the kind introduced in End-to-End Neural Speaker Diarization with Permutation-Free Objectives. That paper reports 12.28% DER (diarization error rate) on simulated mixtures, 31.01% unadapted and 23.07% adapted on CALLHOME, and 11.53% for its x-vector clustering baseline on CALLHOME. We encode the diarizer's per-speaker activity probabilities as tokens of the same count and dimension as our mode tokens, and train with the identical recipe. Beating it requires a gain on attribution and reasoning tiers, not just on discrimination tasks such as speaker counting.
For the second stage, the detector is compared against the same diarizer's overlap and change outputs. Our detector uses no training labels, so matching the diarizer on overlap F1 on CALLHOME would already be informative.
Four ablations isolate the key ingredient. The first trains the same adapter on unprojected encoder features. The second uses a random $k$-dimensional projection instead of the speaker subspace. The third keeps the speaker coordinates but drops $p$ and $\theta$. The fourth uses a single window size instead of the multi-scale signature, which tests whether the switch-versus-superposition test matters.
Why not just…?
Why not just feed full WavLM features to the speech language model? WavLM already trains against overlap and sets results on 14 of 15 SUPERB tasks, including 0.383% EER (equal error rate) on VoxCeleb1. But speech language models built on comparable encoders still fall to 14.9% paired accuracy on HEAR, so encoder quality alone does not reach the reasoning layer. Our first ablation, the adapter on unprojected features, tests this directly. If it matches the projected version, the subspace story adds nothing beyond being a compression trick, and we will say so.
Why not just diarize first and prompt the model with speaker turns? This is a real threat, because a strong diarizer already labels who spoke when, overlap included. The weakness of diarization labels is that they are permutation-ambiguous slot indices with no voice content. A voice-swap counterfactual that leaves timing intact may leave the labels unchanged. Diarization also carries the sim-to-real gap documented by the 31.01% unadapted CALLHOME DER. The diarization-token baseline, trained under the identical recipe, settles the question. If it ties us on HEAR's attribution tier, our contribution reduces to the training-free detector and the geometric analysis.
Why not just keep A2R's transcription-first reasoning? It works, as its 60.5% paired accuracy shows, and it is the bar we must approach. Its cost is that the model must write a full verbatim, speaker-tagged transcript before answering, which its authors identify as unsuitable for streaming use. We compare accuracy against measured latency on the same hardware. A version of our model that is allowed a transcript serves as an upper-bound check.
Why not just fine-tune on CASH without any new input? Perhaps the counterfactual data, not the geometry, does the work. Our "data-only" run uses CASH with direct-answer reward and no mode tokens, at an equal budget. If it matches the full system, the geometric signal is redundant.
Proposed Experiments and Decisive Checks
The geometric check comes first and can kill the project cheaply. On controlled mixtures, $p$ must decrease monotonically as relative loudness approaches 0 dB in at least one layer. At that same layer, the multi-scale test must separate overlap from turn change on held-out AMI and CALLHOME better than a single-scale threshold. We also run the cross-speaker transfer check used in Self-supervised Predictive Coding Models Encode Speaker and Phonetic Information in Orthogonal Subspaces: a subspace fitted on training speakers must still yield clean $p$ and $\theta$ curves on unseen speakers.
The attribution checks are paired accuracy on HEAR overall, on its overlap subset, and per tier. We also report the zero-shot transfer tasks HEAR uses to test A2R: What Do You Like?, where A2R reports +36.4% relative over its base; Find the Spy, where A2R reports +19.0%; and Gaokao listening.
To address the synthetic-to-real gap that A2R leaves open, we build a small attribution question set from real AMI and LibriCSS overlap regions, with answers taken from the human speaker annotations. On this set, the gap between our model and the CASH-only model must be at least as large as on HEAR. The final check is speaker-count scaling on mixtures with two to four concurrent talkers, the regime where On Speaker Attribution with SURT degrades. The rank-2 analysis is expected to strain here, since $\lambda_2$ alone cannot distinguish three voices from two. We will report whether extending the measure to $\lambda_3$ recovers performance, or whether polarization degree only resolves pairwise overlap.
References
- Sudden Polarization Angle Jumps of the Repeating Fast Radio Burst FRB 20201124A
- Self-supervised Predictive Coding Models Encode Speaker and Phonetic Information in Orthogonal Subspaces
- WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing
- A New Emission Mode of PSR B1859+07
- HEAR Who Said What: Unlocking Speaker-Attributed Reasoning via Counterfactual Voice Grounding
- End-to-End Speaker-Attributed ASR with Transformer
- On Speaker Attribution with SURT
- Evaluating the Effects of Embedding with Speaker Identity Information in Dialogue Summarization
- End-to-End Neural Speaker Diarization with Permutation-Free Objectives