Modern voice-conversion systems no longer copy a voice from one summary vector. They read timbre (the "color" of a voice) directly from individual frames of a reference recording. For each sound they must produce, they look up how the target speaker made similar sounds. That lookup is what makes them sound like the speaker. It is also a liability, because a frame-level lookup cannot tell a speaker's nasal resonance from a microphone's resonance, or a breathy fricative from codec hiss. We propose to import a habit from experimental physics. There, a measured quantity counts as real only if it comes out the same when observed through two detection paths with different instrument errors. We apply this both as a test and as a training signal for frame-level speaker representations. Concretely, if the same utterance is captured by two different microphones or codecs, the model's retrieved timbre should be identical for every sound it synthesizes, even though it is free to vary from sound to sound.
The concrete claim is about where invariance should be enforced. Prior work either erases channel information from a single pooled vector, or trains the decoder to output clean speech from a corrupted reference. We instead enforce agreement on the query-conditioned timbre trajectory itself: the sequence of retrieved speaker features, one per output frame. The comparison is made across paired channel views that were chosen to have different systematic errors. We expect this to remove channel contamination at the point where it enters the model, while keeping the phone-by-phone variation that makes dynamic representations worth having. We also expect it to transfer to recording channels never seen in training better than reconstruction-based augmentation does.
What the two directions contribute
In the physics papers the second channel is a check, not just extra data. Daya Bay re-measured the neutrino mixing angle using neutron capture on hydrogen instead of gadolinium. Hydrogen capture is a path with a different energy scale, different backgrounds, and different selection cuts. The two results agreed, which is what licensed combining them (An et al.). MicroBooNE compared muon rates read out through ionization charge and through scintillation light in the same detector (Measurement of the atmospheric muon rate with the MicroBooNE Liquid Argon TPC).
Two lessons carry over. First, agreement validates only if the channels' errors are not shared. The file lists correlated instrument systematics as the main failure mode of the method. Second, neither channel is treated as ground truth. Each is an imperfect observation of the same underlying quantity.
On the speech side, several systems replace the single speaker vector with frame-level retrieval. SEF-VC feeds reference spectrogram frames, with no positional information, into cross-attention queried by content tokens (SEF-VC). TPSE-VC adds a phoneme-to-style attention path beside a global vector (Two-Pathway Style Embedding for Arbitrary Voice Conversion). MTCR-VC retrieves salient time windows and frequency bands under an x-vector query (Multi-Level Temporal-Channel Speaker Retrieval for Zero-Shot Voice Conversion). TVTSyn synchronizes content tokens with a memory of timbre "facets" and keeps the trajectory on a hypersphere (TVTSyn).
These papers motivate the proposal. Their gains come precisely from reading unpooled frames: SEF-VC reports a speaker-similarity score of 0.825 against 0.770 for embedding-based YourTTS on LibriTTS. But all of them were trained and evaluated on clean read speech. MTCR-VC's "channel" means a spectrogram frequency band, not a recording path. None tests what happens when the reference was recorded through a phone codec or a distant microphone.
The constraining work comes from speaker recognition. Channel adversarial training for cross-channel text-independent speaker recognition (CAT) shows that codec mismatch between Speex and SILK hurts pooled embeddings substantially. Its plain CNN reached 69.5% top-1 recall, against 76.4% for the adversarially trained model. CAT removes channel information from a pooled vector with a gradient-reversal layer, a component that flips the sign of a channel classifier's gradient so the encoder learns to hide the channel. It never touches frame-level representations. It also gives no evidence on whether erasing channel cues adversarially would also erase formant transitions and other fine detail. Earlier warnings that normalization layers strip prosody along with nuisance information (Disong Wang et al., Ju-Chieh Chou et al.) suggest it would.
The closest multi-channel speech work uses cross-channel attention across microphone arrays for diarization (Semi-Supervised Multi-Channel Speaker Diarization With Cross-Channel Attention). It uses the channels for spatial separation, not as independent checks on a timbre estimate.
Core Claim
Let $R$ be a reference utterance and let $C_a, C_b$ be two channel transforms with deliberately different error types. Examples are a band-limiting codec versus a reverberant far-field response, or two physical microphones. Let $q_{1:T}$ be the content queries of the source utterance. Let $r_t(\cdot)$ be the speaker feature the model retrieves for output frame $t$, normalized to unit length. We add a channel-swap consistency loss:
$$ \mathcal{L}_{cc} = \frac{1}{T}\sum_{t=1}^{T} \frac{1 - \cos\big(r_t(C_a(R), q), \, r_t(C_b(R), q)\big)}{1 - \overline{\cos}_{\text{other}}(t)} $$
Here $\overline{\cos}_{\text{other}}(t)$ is the mean cosine similarity between $r_t$ and the trajectories retrieved, for the same query frame, from references of other speakers in the batch.
The denominator stops the model from satisfying the loss by collapsing to a constant trajectory. Agreement only counts as agreement relative to how different speakers look at the same phonetic position. The usual reconstruction loss is unchanged. Because the comparison is made per query frame, the trajectory may still differ between a vowel and a fricative. It is not allowed to differ between two recordings of the same vowel.
The hypothesis has three parts. Trajectory-level agreement will reduce channel leakage into converted speech. It will preserve per-context timbre variation better than adversarial erasure. And it will outperform reconstruction-only augmentation specifically on channels held out from training. The third part is the one that could fail, and the experiments are built around it.
Research Program
The first component is a measurement study, which we expect to be useful whatever happens to the method. We define a channel-swap test for any reference-based converter. The test converts the same source with the same reference utterance seen through two channels, then asks three questions. Do the two outputs differ in speaker similarity? Can a channel classifier trained on natural speech identify the reference's channel from the converted output? Which phone classes carry the discrepancy? This mirrors per-channel comparison in physics, and it shows where frame-level retrieval is vulnerable. Our prediction is that leakage concentrates in high-frequency classes such as fricatives and sibilants under codecs, and in vowels under reverberation.
The second component is the training method above. It is built on a reimplementation of the SEF-VC backbone, chosen because its bag-of-frames attention is the most exposed to frame-level contamination. The trajectory $r_t$ is the output of its reference cross-attention blocks. A second instantiation on a TVTSyn-style facet memory checks that the result is not specific to one architecture. In that version, consistency is imposed on the gated facet mixture before spherical interpolation, which also addresses the file's open question of how channel shift distorts those trajectories.
The third component takes the physics warning about correlated errors seriously. We train with channel pairs whose errors overlap, such as Speex versus SILK at similar bitrates, and with pairs whose errors are orthogonal, such as codec versus reverberation. We then measure which choice generalizes to unseen channels. If the physics intuition holds, orthogonal pairs should win even at equal augmentation diversity.
Baselines
The naive reference point is the SEF-VC-style backbone trained on clean LibriTTS and tested with channel-corrupted references. On clean LibriTTS test speakers, SEF-VC reports a speaker-similarity score of 0.825, a mean opinion score of 4.37 and a character error rate of 5.53%. We found no reported result on Lacuna for any frame-level voice-conversion system under codec or microphone mismatch. That missing number is part of what the measurement study supplies. We re-measure the backbone on our clean split first, so that degradations are read against our own reproduction.
The strongest competitor is the same backbone trained with multi-condition augmentation: references are passed through random channels, and the target remains the clean utterance. It uses exactly the same channel pool, training hours, update count and hyperparameter search budget as our method. It is the strongest competitor because reconstruction from a corrupted reference already pressures the network to ignore channel effects, so it is the natural explanation for any gain we see.
The second competitor is CAT transplanted to frame level. A gradient-reversal channel classifier is attached to the retrieved trajectory with the reversal weight set to 1, the value CAT found best, plus a small search around it. CAT's own reported figures (76.4% top-1 recall, 5.8% EER) come from a proprietary corpus we cannot use, so they serve only as evidence that the mechanism helps pooled vectors, not as a target. A pooled-vector version of CAT feeding a global-conditioning decoder is also run. This isolates whether frame-level retrieval is worth its exposure to channels at all.
For cross-dataset context we report MTCR-VC's published out-of-dataset speaker MOS of 3.68 and accuracy of 95.8% on VCTK, CMU Arctic and HiFi-TTS (Multi-Level Temporal-Channel Speaker Retrieval for Zero-Shot Voice Conversion). These were measured on clean references, and we treat them only as a sanity check.
The key ablation removes the denominator in $\mathcal{L}_{cc}$, which tests for collapse. A second ablation applies consistency to the final output rather than to the trajectory, which tests whether the location of the constraint matters.
Beating the strongest baseline means the following. On held-out channel families, our method has lower channel-classifier accuracy on outputs and higher speaker similarity to a clean enrollment of the target. These gains must hold across three seeds with non-overlapping intervals, with no worse character error rate, and without reducing per-phone-class timbre variance below the clean backbone's.
Why not just…?
Why not just use a static speaker vector with channel adversarial training? The file's evidence is that static vectors lose phone-level detail: YourTTS reaches 0.770 against 0.825 for SEF-VC's frame-level attention, and TPSE-VC beats global-embedding converters. CAT fixes the channel problem but not this averaging problem. We run the pooled-vector CAT baseline under our channel-mismatch protocol anyway. If it matches our method on speaker similarity under mismatch, that would show frame-level retrieval loses its advantage once references are dirty, which would be a useful finding in itself.
Why not just augment the references and keep a clean target? This is the real threat. It may well match our method on channels seen in training. Our argument is mechanistic. A reconstruction loss lets the decoder repair channel effects after the retrieval step has already read artifact frames, and nothing forces that repair to generalize to a new kind of distortion. The decisive comparison is therefore on held-out channel families, together with direct inspection of attention mass on artifact-heavy frames, such as codec-damaged high bands or reverberant tails. If augmentation is just as good on held-out channels, we will report that. The contribution then shrinks to the channel-swap test and the orthogonal-pairing result, both of which also apply to the augmentation baseline.
Why not just rely on instance normalization or a vector-quantized content bottleneck? Those components act on the content path, not on the speaker path. Prior work reports that they strip prosody and pronunciation detail (Disong Wang et al., TVTSyn). The TVTSyn-style instantiation already includes a quantized content bottleneck, so the channel-swap test on it directly shows whether that bottleneck alone stops channel effects entering the timbre lookup.
Why not just apply gradient reversal at frame level? That is our second baseline. Its specific risk is that a frame classifier can detect a codec from genuine high-frequency speaker cues, so reversing its gradient removes those cues as well. We measure this through per-phone-class timbre variance and fricative-specific speaker similarity, where we expect adversarial erasure to cost the most.
Proposed Experiments
Training uses the LibriTTS subset described for SEF-VC (about 586 hours, 2,311 speakers). Synthetic channels are drawn from Speex and SILK at several bitrates (the codecs studied by CAT), microphone frequency-response curves, and room impulse responses. One family (reverberation in one split, codecs in another) is held out entirely for testing.
Real paired channels come from VCTK, whose public release provides each utterance from two microphones. These pairs are used only at test time, so the channel-swap test is run on physically recorded pairs no synthetic transform produced. That is the closest speech analogue to two detector readouts of the same event. Unseen-speaker protocols follow SEF-VC on the LibriTTS test-clean speakers and TPSE-VC's 20 unseen VCTK speakers with 2,000 pairs.
Metrics are as follows. Speaker similarity is the cosine similarity of verification embeddings between the output and a clean enrollment of the target. Verification equal error rate is computed against the same enrollment. Character error rate measures intelligibility. Channel leakage is the accuracy of a channel classifier on converted outputs. Channel-swap divergence is the embedding distance between outputs produced from the two channel views, divided by the within-speaker distance on clean speech, and it is broken down by phone class. Per-phone-class trajectory variance guards against the model flattening its timbre. A small listening test covers naturalness and similarity.
The decisive check is a single comparison: our method against augmentation on held-out channel families and on real VCTK two-microphone pairs, at matched data and compute. Two secondary checks follow. The orthogonal-versus-correlated pairing experiment tests whether the physics requirement of independent errors carries over to learned representations. The phone-class breakdown tests whether gradient reversal at frame level removes speaker detail in exactly the contexts where frame-level retrieval had its advantage. We will treat trajectory-level consistency as justified only if it wins the first comparison without losing the third.
References
- Independent measurement of the neutrino mixing angle<mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML" display="inline"><mml:mrow><mml:msub><mml:mrow><mml:mi>θ</mml:mi></mml:mrow><mml:mrow><mml:mn>13</mml:mn></mml:mrow></mml:msub></mml:mrow></mml:math>via neutron capture on hydrogen at Daya Bay
- Measurement of the atmospheric muon rate with the MicroBooNE Liquid Argon TPC
- SEF-VC: Speaker Embedding Free Zero-Shot Voice Conversion with Cross Attention
- Two-Pathway Style Embedding for Arbitrary Voice Conversion
- Multi-Level Temporal-Channel Speaker Retrieval for Zero-Shot Voice Conversion
- TVTSyn: Content-Synchronous Time-Varying Timbre for Streaming Voice Conversion and Anonymization
- Channel Adversarial Training for Cross-channel Text-independent Speaker Recognition
- VQMIVC: Vector Quantization and Mutual Information-Based Unsupervised Speech Representation Disentanglement for One-Shot Voice Conversion
- Diffusion-Based Voice Conversion with Fast Maximum Likelihood Sampling Scheme
- Semi-Supervised Multi-Channel Speaker Diarization With Cross-Channel Attention