Salient Speech Representations Based on Cloned Networks

W. Bastiaan Kleijn, Felicia S. C. Lim, Michael Chinen, Jan Skoglund

In speech processing, the goal of feature extraction is to represent a signal using a compact set of "meaningful" attributes. However, traditional unsupervised methods like Variational Autoencoders (VAEs) often struggle to distinguish between perceptually relevant speech content and irrelevant background noise. This paper, presented at INTERSPEECH 2019 by researchers from Google and Victoria University of Wellington, introduces a novel "cloned network" framework designed to extract salient features that are invariant to noise and distortions.

The Challenge of Saliency and Disentanglement

Standard unsupervised models often fail to achieve disentanglement—the ability to separate independent factors like pitch, phonemes, and background noise into distinct feature dimensions. Because raw waveforms contain significant amounts of perceptually irrelevant information (such as the specific phase of noise), VAEs tend to waste "bitrate" on details that do not matter to a human listener.

The authors address this by defining "salient" features as those that remain consistent across different versions of the same underlying information (e.g., the same sentence spoken in a quiet room versus a noisy cafe). By forcing a model to ignore variations that the designer deems irrelevant, they provide a "weak form of supervision" to guide the learning process.

Proposed Methodology: Cloned Networks

The core innovation is the Cloned Network training setup, which utilizes multiple identical copies of an encoder-decoder architecture.

The Cloned Network architecture uses shared-weight encoders to process multiple equivalent inputs, optimized via similarity and independence losses.

Architecture and Training

  1. Weight Sharing: The system employs $Q$ identical clones of an encoder network. During training, all clones share the same weights.
  2. Equivalent Inputs: Each clone is fed a different "equivalent" signal. For instance, Clone 1 receives a clean speech snippet, while subsequent clones receive the same snippet corrupted by different types of background noise.
  3. Objective Function: The loss function consists of three primary components:
  4. Similarity Loss: Forces the feature vectors from all clones to be as close as possible, teaching the network to ignore noise that varies across clones.
  5. Independence (MMD): Uses Maximum Mean Discrepancy (MMD) to ensure features follow a specific distribution (e.g., i.i.d. Laplacian), encouraging sparsity and disentanglement.
  6. Optional Reconstruction: A shared decoder can map features back to a clean target signal to ensure the representation retains sufficient information for high-quality synthesis.

Experimental Setup

The model was evaluated using both a synthetic toy dataset and real-world speech data. The toy dataset was designed to simulate "whispered speech" to test if the model could disentangle non-linear spectral peaks (formants) from excitation noise. For real-world testing, the authors used the WSJ0 (Wall Street Journal) database combined with a noise corpus of approximately 10,000 diverse environmental recordings.

To demonstrate the utility of these features, the researchers used 12 extracted salient features as local conditioning for a WaveNet vocoder. This allowed the system to function simultaneously as a speech coder (compressing speech into 12 dimensions) and an enhancement system.

Key Results

The toy experiments confirmed that the cloned approach successfully disentangled highly non-linear features without requiring a decoder. In real-world speech tasks, the system demonstrated remarkable robustness and quality.

MUSHRA scores showing that the SalientS-dw model maintains high subjective quality even under noisy conditions, significantly outperforming PCA baselines.

Subjective listening tests (MUSHRA) revealed that the clone-based system (SalientS-dw) significantly outperformed standard Principal Component Analysis (PCA) baselines. Notably, when presented with noisy input (0-10 dB SNR), the system produced speech quality nearly identical to its performance on clean input. This suggests the encoder effectively "stripped away" noise, providing only the salient speech information to the WaveNet reconstructor.

Conclusion and Future Work

The Cloned Network framework provides a powerful method for designer-defined saliency. While the current work focused on a low-dimensional representation (12 features) and specific noise types, it proves that "equivalence" can serve as a robust training signal for unsupervised feature extraction. Future research may explore how this scales to higher-dimensional latent spaces and more complex tasks where the definition of "equivalence" is harder to specify.

Summary updated

BibTeX
@inproceedings{kleijn2019salient,
  author = {W. Bastiaan Kleijn and Felicia S. C. Lim and Michael Chinen and Jan Skoglund},
  title = {{Salient Speech Representations Based on Cloned Networks}},
  booktitle = {Interspeech},
  year = {2019},
  doi = {10.21437/interspeech.2019-1861},
}
Figure