Speaker recognition systems that identify people reliably from their speaking voice largely fail when the same person sings, and they also fail when the recording is made far from the microphone in a reverberant room. These two failures are usually studied separately. In real data they are tangled together. Singing is typically captured dry in a studio and then drenched in artificial reverb effects, while speech is captured in ordinary rooms. A model comparing a spoken clip to a sung clip is therefore facing two changes at once: a change in how the voice is produced and a change in the acoustic environment. Nobody has measured how much of the speech-to-singing identity gap is really an acoustics gap, and no training recipe separates the two.
We propose to separate them with simulation. We take the few corpora where the same people both speak and sing, and re-render every clip through many simulated acoustic conditions. These include physical rooms, studio reverb plugins, and binaural head-and-ear positions. We then train a speaker embedding with a triplet objective whose negatives are deliberately matched to the positive on every nuisance factor: same vocal mode, same room, same position, different person. Such a model cannot tell people apart by room or by singing-versus-speaking style, because those cues never separate a positive from its negative. A triplet objective, in one sentence, takes an anchor clip, a clip that should land close to it, and a clip that should land far, and pushes the embedding to respect that ordering by a margin. The novelty is not the triplet loss itself. It is the combination of simulation that breaks the natural correlation between vocal mode and acoustics, a negative-selection rule that exploits that control, and a factorial test set that reports how much of the cross-modal gap is acoustic.
Core Claim
A measurable share of the speaker-verification error between speech and singing comes from acoustic mismatch, not from vocal production. That share can be removed by training on renders where mode and acoustics are decorrelated, with negatives matched on nuisance factors. The residual gap, measured under matched acoustics, is the true vocal-production gap. The claim can be falsified in two ways. If cross-modal error barely changes between matched-acoustics and mismatched-acoustics trials, acoustics is not a meaningful part of the gap. If a standard classification-loss speaker model trained on the same decorrelated renders matches our method, then the contribution is the data construction alone, and we will report it that way.
Relation to Prior Work
Three papers motivate the proposal by each solving one slice of the problem in isolation.
Bridging Speech and Singing aligns speech and singing speaker embeddings of the same person with a cosine embedding loss, as a component of speech-prompted singing conversion. It works only on dry studio audio and evaluates generation, not verification. It reports speaker similarity 0.506 versus 0.448 for its CoMoSVC baseline.
Delving into VoxCeleb removes recording-environment cues from speech speaker embeddings. It does this with an adversarial critic trained by triplet loss, using YouTube video ID as a coarse stand-in for "environment." It reports EER falling from 13.14% to 10.56% on a replayed test set. We replace the video-ID proxy with exact, simulated environment labels, and we extend the idea to singing.
Spatialization Quality Metric for Binaural Speech builds triplets whose anchor and positive share a spatial position but differ in spoken content. That is the mirror image of our construction: they isolate position from content, and we isolate identity from position, room, and mode. Their work covers speech only.
Two papers supply the simulation machinery. FRA-RIR generates physical-room impulse responses about 110 times faster than image-source tools and without their "sweeping echo" artifacts, which makes on-the-fly rendering per batch feasible. ReverbFX captures 1,846 impulse responses from music-production reverb plugins, with decay times up to 52 seconds. It shows that physical-room training transfers poorly to produced vocals, reaching POLQA 2.53 versus 3.12 in-domain for dereverberation.
That ReverbFX result also constrains us. A simulator built only from physical rooms, such as Spatial LibriSpeech or VAST, would likely miss the acoustics that singing actually comes with. We therefore mix both kinds.
The spatial-audio literature is speech-only, which is why this project matters for spatial audio. No spatial singing corpus exists, so any spatially robust singer embedding must be trained on simulated data.
On the human side, Familiar and Unfamiliar Speaker Identification in Speech and Singing finds that unfamiliar listeners score best on speech-to-speech matching (over 57%) and worse across modes. That study was run on dry audio. It motivates the question but does not tell us how room acoustics interact with vocal mode.
Research Program
The training material comes from singing and speech corpora in the file. For same-person speech and singing pairs we use GTSinger, which has 16.16 hours of paired speech in which singers speak their lyrics, and NUS-48E. For singing-only data we use M4Singer and OpenSinger. For speech-only data we use Aishell-3 and VoxCeleb1. Speakers are split so that test identities are never seen in training.
Each clip $x$ is rendered on the fly as $x * h$. The impulse response $h$ is drawn from one of three pools. The first is FRA-RIR physical rooms. The second is ReverbFX plugin responses. The third is a binaural pool formed by combining FRA-RIR reverberation tails with public head-related responses at sampled directions and distances, so that spatial position becomes a controlled nuisance factor.
The objective is a triplet loss with factor-matched negatives. Let $f$ be the speaker encoder, $d$ cosine distance, and $\alpha$ the margin. Let $x^{s,m}$ be a clip of speaker $s$ in vocal mode $m$ (speech or singing), and $h_1, h_2$ two independently drawn impulse responses. The loss is
$$ \mathcal{L} = \max\Big(0,\; d\big(f(x_a^{s,m} * h_1),\, f(x_p^{s,m'} * h_2)\big) - d\big(f(x_a^{s,m} * h_1),\, f(x_n^{s',m'} * h_2)\big) + \alpha\Big), $$
where $m'$ is usually the other mode than $m$, and $s' \neq s$. The negative shares the positive's vocal mode and its exact impulse response. Any shortcut through room, reverb plugin, position, or singing style therefore gives zero advantage. Within a batch we mine the hardest negative among different speakers rendered with the same $h_2$ and mode $m'$. This is how we address the known problem that triplet training is sensitive to how negatives are chosen.
The encoder is a ResNet-34 speaker network of the kind used by Huh et al., trained from scratch. For comparability we also fine-tune the ESPnet-SPK embedding used by SSAN.
Evaluation uses a factorial trial list on held-out speakers. Enrollment and test clips are crossed over mode (speech-speech, speech-singing, singing-singing) and acoustics (identical render, different physical room, physical versus plugin, different binaural position). This design lets us report EER in each cell. It also lets us define the acoustic share of the cross-modal gap: cross-modal EER under mismatched acoustics minus cross-modal EER under identical renders, relative to the total cross-modal gap over dry speech-speech.
Simulated tests can flatter a simulation-trained model, so we also record a small real test set. It follows the replay protocol of Huh et al.: held-out GTSinger and NUS-48E speech and singing are played through a loudspeaker in two or three real rooms and captured with a binaural dummy head. This is the first real spatial speech-and-singing verification set, and it will be released.
Baselines
The naive reference is the pretrained ESPnet-SPK speaker embedding applied with no adaptation. It calibrates how bad the problem is in each factorial cell. There is no reported result found on Lacuna for cross-modal speech-versus-singing speaker-verification EER on GTSinger or NUS-48E, so this baseline must be measured by us.
The strongest competitor is a ResNet-34 trained with a standard additive-margin softmax classification loss. It sees exactly the same corpora, the same three impulse-response pools, and the same on-the-fly renders, but with rooms drawn independently per clip. It is the strongest because it has every data advantage our method has. The only difference is that it lacks the explicit rule that negatives share the positive's room and mode. It receives the same GPU hours and the same hyperparameter search budget (margin, learning rate, loss scale) on a speaker-disjoint validation split.
The adversarial environment-invariance method of Delving into VoxCeleb is reimplemented as a third baseline. Its environment label is the exact simulated render ID, which is strictly better information than the video IDs of the original. Its reported numbers are an EER of 5.26% on VoxCeleb1 and 10.56% on the replayed set, both on speech only. These are useful anchors for the speech-speech cells, not for singing.
The SSAN-style cosine alignment of speech and singing embeddings is trained on dry audio and then tested on rendered trials. This measures what cross-modal alignment alone buys when acoustics change.
A dereverberation front end completes the external baselines. SGMSE+ trained on SingingReverbFX, which reports POLQA 3.12 for studio vocals, is placed before the dry-trained model.
Beating a baseline means lower EER on the mismatched-acoustics cross-modal cells and on the real replayed set. The improvement must exceed the 95% bootstrap interval computed by resampling speakers, and it must come without a worse result on dry speech-speech trials.
The key ablations isolate our ingredient. The first is the same triplet loss with random negatives rather than factor-matched ones. The second is factor-matched negatives with physical rooms only, without ReverbFX. The third is factor-matched negatives without the binaural pool.
Why not just…?
Why not just train a normal speaker model on pooled speech and singing with heavy reverb augmentation? This is the real threat, and it is the main baseline above. Augmentation randomizes rooms, but it never forces the model to separate two different people heard in the identical room and mode. A classification loss can still partly lean on acoustic and stylistic cues that correlate with identity in the training corpora; for example, each singing corpus has its own studio. Our decisive comparison runs this model on identical renders. If it matches the factor-matched triplet model on the mismatched cells and the replayed set, we conclude that decorrelated simulation is sufficient and that the negative-selection rule adds nothing. That would still leave the factorial benchmark and the acoustic-share measurement as the contributions.
Why not just dereverberate first and then use a dry cross-modal model? ReverbFX shows generative dereverberation can recover produced vocals reasonably well. However, SingFake found that a separation front end did not reliably help singing deepfake detection, because processing artifacts corrupt fine vocal cues. The same risk applies to identity cues. The front-end baseline tests this directly on the plugin and binaural cells. Dereverberation also cannot undo a change in spatial position, which the binaural cells probe.
Why not just make speech sound like singing with pitch and duration augmentation? PDAugment helps lyrics transcription this way. However, pitch-shifted speech lacks the vibrato, register changes, and altered phonation modes that Analysis and Classification of Phonation Modes In Singing identifies. It also leaves the acoustics confound untouched. We include pitch and duration augmentation of the speech side as an optional addition to the strongest baseline, so its gain is measured rather than assumed.
Decisive Checks
Four checks decide the claim. The first is the acoustic-share measurement for the naive and strongest baselines: if acoustic mismatch accounts for little of the cross-modal EER gap, the premise fails and we stop there. The second is the factor-matched versus random-negative ablation on identical training data, which isolates the proposal's single new ingredient. The third is transfer to the real replayed binaural set, which guards against the model merely learning the simulator. The fourth is a linear probe that predicts the impulse-response pool, room, and binaural direction from frozen embeddings. Lower probe accuracy for our model than for the baselines, at equal verification accuracy on dry speech-speech trials, would show that the gain comes from discarding acoustic information rather than from an accident of tuning.
A secondary check tests simulator fidelity. We re-render with omnidirectional versus speech-directivity sources. If results are sensitive to directivity, that points to singing-specific directivity modeling as the next bottleneck for spatial singing data.
References
- Bridging Speech and Singing: Multi-stage Speech-Prompted Singing Voice Conversion with Speaker Embedding Adaptation
- Delving into VoxCeleb: Environment Invariant Speaker Recognition
- Spatialization Quality Metric for Binaural Speech
- FRA-RIR: Fast Random Approximation of the Image-source Method
- ReverbFX: A Dataset of Room Impulse Responses Derived from Reverb Effect Plugins for Singing Voice Dereverberation
- Spatial LibriSpeech: An Augmented Dataset for Spatial Audio Learning
- VAST: The Virtual Acoustic Space Traveler Dataset
- Familiar and Unfamiliar Speaker Identification in Speech and Singing
- GTSinger: A Global Multi-Technique Singing Corpus with Realistic Music Scores for All Singing Tasks
- SingFake: Singing Voice Deepfake Detection
- PDAugment: Data Augmentation by Pitch and Duration Adjustments for Automatic Lyrics Transcription
- Analysis and Classification of Phonation Modes In Singing