A neural vocoder turns a compact description of speech, usually a mel-spectrogram (a time-frequency picture of the sound's energy), into an audible waveform. Source-filter vocoders build in the physics of the voice. A "source" signal stands for the buzz of the vocal folds and is driven by pitch. A "filter" stands for the shaping done by the throat and mouth. These models sound excellent on studio recordings. They break when the input was recorded on a phone in a reverberant kitchen, because the room and the microphone are also filters, and they end up mixed into the same features the vocoder reads as "vocal tract." We propose to give the model a third, explicit filter for the room and microphone. We then use adversarial domain adaptation to make that split learnable from unpaired recordings. Adversarial domain adaptation is a training trick in which a small classifier tries to guess which recording condition a feature came from, and the feature extractor is penalized whenever it succeeds.
The core idea fits in two sentences. Speech is modeled as source × vocal tract × channel, where "channel" means the room and microphone together. An adversary forces the vocal-tract features to be indistinguishable across recording conditions, so all condition-specific information has to flow through the explicit channel branch. The adversary does more than clean up features. It is what makes the dry/wet factorization identifiable without paired clean and reverberant recordings. The payoff is a vocoder that can resynthesize studio-dry speech from degraded input, and can put back a chosen room or microphone, trained only on clean data plus unlabeled-by-pairing target-domain audio.
Core Claim
Reverberation Modeling for Source-Filter-Based Neural Vocoder already convolves a dry source-filter output with a learned room impulse response, which is the room's echo signature as a filter. It also predicts that response per utterance for unseen rooms. That paper is the method we reuse. It is also the gap that motivates us. Its training is supervised on dry and reverberant targets, and the authors note that dry synthesis from reverberant features stays flawed because the reverberation is "baked into" the input features. Nothing in their objective stops the vocal-tract path from carrying room information. Our claim is that a gradient-reversal domain adversary on the vocal-tract path alone closes this loophole. This is the gradient-reversal mechanism that W2VC used to strip speaker identity from WavLM content features. W2VC applied it to a monolithic content vector feeding a black-box HiFi-GAN, on clean VCTK and ARCTIC. It used the adversary for speaker removal, not for channel removal, and had no source-filter split. We instead place it at the one point in a physically factored model where condition information should be absent. The claim is falsifiable. If the adversary works as argued, the dry output should match held-out clean references better than the same architecture trained without the adversary. A probe classifier should also fail to recover the recording condition from vocal-tract features while succeeding on the channel embedding.
Mechanism
The analysis encoder $E_f$ maps an input mel-spectrogram $x$ to frame-level vocal-tract features $z = E_f(x)$. An utterance-level channel encoder $E_c$ maps $x$ to a predicted impulse response $h = E_c(x)$, following the time-variant design of the reverberation paper. A source module in the style of SF-GAN turns pitch $F_0$ into a sine-plus-noise excitation. Its resolution-wise filter network produces a dry waveform $d = G(F_0, z)$, and the observed waveform is modeled as $\hat{y} = d * h$. A domain discriminator $D$ predicts the recording condition $k$ (studio, or one of the target conditions) from $z$ through a gradient reversal layer. Training minimizes
$$ \mathcal{L} = \mathcal{L}_{\text{rec}}(\hat{y}, y) + \mathcal{L}_{\text{GAN}}(\hat{y}) + \mathcal{L}_{\text{dry}}(d, y)\,\mathbb{1}[k=\text{studio}] - \lambda\, \mathcal{L}_{\text{CE}}(D(z), k), $$
where $y$ is the observed waveform, $\mathcal{L}_{\text{rec}}$ is a multi-resolution spectral reconstruction loss, $\mathcal{L}_{\text{GAN}}$ is the usual vocoder adversarial loss, $\mathcal{L}_{\text{dry}}$ applies only to studio data (whose channel is constrained toward an identity impulse), $\mathcal{L}_{\text{CE}}$ is the discriminator's cross-entropy, and $\lambda$ is the reversal weight.
The identifiability argument works as follows. Studio data pins $G$ to produce clean speech from studio-distributed $z$. The adversary pushes target-domain $z$ into that same distribution. So $d$ cannot carry reverberation without the discriminator detecting it, and the only remaining way to explain the observed wet audio is through $h$.
The obvious failure mode is leakage: the encoder could smuggle room information into $F_0$. We guard against it in two ways. Pitch comes from a fixed tracker, not a learned path. A second, optional reversal head on the excitation tests whether this matters.
A second failure mode is over-alignment. FAN-Net found that normalizing more than about 10% of Fourier amplitudes destroyed boundary information. This constrains us. We sweep $\lambda$ and report phonetic intelligibility alongside alignment, because a $z$ that has lost phonetic detail would also fool the discriminator.
Research Program
We build on DDS, described in DDS: A new device-degraded speech dataset for speech enhancement. DDS re-recorded clean VCTK and DAPS speech in 27 conditions: 9 environments, 3 microphones including a MEMS smartphone microphone, and several positions. DDS is paired, which we exploit only for evaluation and for an oracle. For training the proposed model, target-condition audio is used without its clean counterparts, and clean studio audio comes from disjoint speakers and utterances. This way no utterance is ever seen in both forms. We hold out entire environments and one microphone type to measure generalization to unseen conditions. We additionally test on the 28-speaker, 18-condition reverberant database from the reverberation paper, using its T1/T2/T3 splits, so that we can compare directly with its time-variant impulse-response model. The analysis program has three parts. First, we train linear probes for condition from $z$, $h$, and the excitation. Second, we swap channels between utterances: we pair $z$ from one recording with $h$ from another and check that the room character transfers while the content does not. Third, we break results down by condition type.
Baselines
The naive reference point is an SF-GAN-style vocoder trained only on studio data and applied directly to degraded mel-spectrograms. It shows how badly baked-in channel effects hurt. SF-GAN reports no statistically significant preference against ground truth on LJSpeech (p = 0.25), but no reported result was found on Lacuna for source-filter vocoders under device or room shift.
The strongest existing source-filter competitor is the time-variant impulse-response vocoder from Reverberation Modeling for Source-Filter-Based Neural Vocoder. We reimplement it on the same SF-GAN backbone so that only the training signal differs. Because it needs dry/wet pairs, we run it in two settings. In the first it uses DDS pairs, which is an oracle our unpaired method should approach. In the second it uses simulated pairs, built from public impulse responses applied to studio data, which is a fair unpaired comparison. Its paper reports MUSHRA gains over HiNet and NSF baselines, but no numerical score was found on Lacuna. Beating the simulated-pair version on held-out real conditions, and closing most of the gap to the oracle, would count as success.
The strongest pipeline competitor is a speech-enhancement front end followed by a clean vocoder. On DDS, the dataset paper reports that MelMapping performed best overall in PESQ and ESTOI, but no specific numerical PESQ/ESTOI score was found on Lacuna. We train MelMapping on the same DDS training conditions and feed its output to the studio-trained vocoder.
Two ablations isolate the key ingredient. The first keeps the explicit channel branch and removes the adversary ($\lambda = 0$). This tests whether the factorization alone is identifiable, and our argument predicts it is not. The second keeps the adversary and removes the channel branch, which is essentially W2VC's gradient reversal transplanted onto the vocal-tract path. Without somewhere for the room information to go, this should either fail to reconstruct wet audio or strip phonetic detail. All systems share data, the backbone, a fixed GPU-hour budget, and an equal number of hyperparameter trials.
Why not just…?
Why not just apply cepstral mean normalization? Subtracting the per-utterance mean of the log spectrum removes any stationary linear filter. A microphone's frequency response is close to exactly that. This is a real threat, and we expect it to roughly match our method on DDS conditions that differ only by microphone in a quiet, low-reverberation space. Reverberation is different. Its impulse response is longer than an analysis frame, so it smears energy across frames instead of adding a constant offset, and mean subtraction cannot undo it. Our condition-type breakdown settles the question. If normalization matches us on reverberant environments too, the adversarial component is not earning its keep, and we would say so.
Why not just augment with simulated rooms and equalization, as NANSY does? NANSY and NANSY++ obtain disentanglement through hand-designed perturbations. The latter does so at a scale of 10,571 hours. DDS was built precisely because synthetic degradation misses real device and room interactions. Our simulated-pair baseline is this alternative. If it matches the adversarial model on held-out real conditions, the case for using unpaired real audio collapses.
Why not just enhance first and then vocode? A cascade is simple and strong. However, the enhancer has no notion of pitch or excitation. Its errors are therefore likely to appear as voicing artifacts in the downstream source module, which SF-GAN identifies as its main sensitivity. A cascade also cannot re-apply a chosen room. The MelMapping cascade baseline measures the quality side directly. We add $F_0$ and voicing error metrics specifically to test the artifact hypothesis.
Why not just fine-tune a black-box vocoder on target audio? A fine-tuned black-box vocoder would learn to reproduce the target room, not remove it. It has no dry output at all. That gives it nothing to offer for dereverberated resynthesis. It does remain a valid upper reference for wet-reconstruction quality, which we report.
Proposed Experiments
The primary metrics compare the dry output with DDS clean references on held-out environments and microphones. They are PESQ and ESTOI for quality and intelligibility, mel-cepstral distortion, $F_0$ RMSE, and a voicing error rate for the source path. We add ASR word error rate as a check on over-alignment, alongside MUSHRA listening tests on a stratified subset. For mechanism, we report probe accuracy for recording condition from $z$, which should be near chance, and from $h$, which should be high. We also report the success of the channel-swap test, measured by the T60 (reverberation time) error of resynthesized wet audio against the donor condition, a measure the reverberation paper also uses.
The decisive checks are three. The first is whether the adversarial model beats both the $\lambda = 0$ ablation and the simulated-pair impulse-response baseline on held-out reverberant DDS environments. If it does not, the identifiability argument is wrong. The second is whether its advantage over cepstral mean normalization is concentrated in reverberant rather than microphone-only conditions. That is the pattern predicted by the convolution argument; a uniform advantage would suggest a different explanation. The third is whether word error rate stays flat across the $\lambda$ sweep while condition-probe accuracy falls. That would show the vocal-tract features lost the room without losing the words.
References
- Reverberation Modeling for Source-Filter-Based Neural Vocoder
- W2VC: WavLM representation based one-shot voice conversion with gradient reversal distillation and CTC supervision
- Source-Filter-Based Generative Adversarial Neural Vocoder for High Fidelity Speech Synthesis
- Fan-Net: Fourier-Based Adaptive Normalization for Cross-Domain Stroke Lesion Segmentation
- DDS: A new device-degraded speech dataset for speech enhancement
- Neural Analysis and Synthesis: Reconstructing Speech from Self-Supervised Representations
- NANSY++: Unified Voice Synthesis with Neural Analysis and Synthesis