Raw waveform-based speech enhancement by fully convolutional networks

Szu-Wei Fu, Yu Tsao, Xugang Lu, Hisashi Kawai

Researchers from Academia Sinica and NICT have proposed a significant shift in speech enhancement (SE) methodology. Traditionally, SE systems process frequency-domain representations of sound, such as magnitude spectrograms. This paper explores an end-to-end alternative: using Fully Convolutional Networks (FCNs) to process raw audio waveforms directly, bypassing the need for Fourier transforms and addressing long-standing issues with phase preservation and high-frequency reconstruction.

The Challenge of Raw Waveforms

Most existing speech enhancement systems do not operate on raw audio. Instead, they convert signals into a magnitude spectrogram (like a Log Power Spectrum), clean the resulting "image," and then reconstruct the audio. This approach introduces two primary flaws:

  1. The Phase Problem: These systems typically ignore the signal's phase, reusing the noisy phase from the original input. This leads to audible artifacts and reduced perceptual quality.
  2. High-Frequency Loss: Standard Deep Neural Networks (DNNs) or Convolutional Neural Networks (CNNs) often struggle to reconstruct high-frequency components when applied to raw waveforms.

The authors identify Fully Connected (FC) layers as the primary culprit. In a standard DNN, every output sample is connected to every node in the previous layer. This global connectivity makes it mathematically difficult for the network to simultaneously learn the rapid changes of high-frequency signals and the slower changes of low-frequency signals. The model often "sacrifices" rare high-frequency patterns to satisfy more common low-frequency patterns during optimization.

Fully Convolutional Architecture

To overcome these limitations, the researchers proposed a Fully Convolutional Network (FCN). Unlike standard CNNs that conclude with dense, fully connected layers, the FCN consists entirely of convolutional layers.

Architecture of the proposed FCN showing local receptive fields (G, F) and weight sharing (R_t) for processing input samples (x_t) to output samples (y_t).

By utilizing only convolutions, each output sample depends only on a local "neighborhood" of input samples. This mimics the behavior of physical filters and allows the network to preserve local temporal structures more effectively. The FCN acts as a sophisticated, learnable filter bank, handling the interdependency between neighboring audio samples more naturally than a dense network.

Experimental Results and Efficiency

The model was trained using the TIMIT corpus, corrupted by five noise types (Babble, Car, Jackhammer, Pink, and Street). To test robustness, it was evaluated against mismatched noise types (White Gaussian, Engine, and Baby crying) that were not present during training.

One of the most striking findings was the model's efficiency. By removing the parameter-heavy FC layers, the FCN achieved superior performance with only 0.2% of the parameters required by traditional DNN and CNN baselines.

Waveform reconstruction comparison: The FCN preserves high-frequency details that are lost or "smeared" in the standard DNN baseline.

In terms of objective metrics, the FCN significantly outperformed the industry-standard LPS-based DNN:

  • Intelligibility (STOI): The FCN achieved a score of 0.722, compared to 0.691 for the LPS-DNN.
  • Quality (PESQ): The FCN reached 2.063, surpassing the LPS-DNN's 1.879.

Visual evidence from waveform analysis confirms that while DNN-based models tend to lose high-frequency speech components, the FCN preserves them, resulting in much clearer audio reconstruction.

Conclusion and Future Directions

The FCN approach demonstrates that raw waveform processing is not only viable but superior to traditional spectrogram-based methods in both quality and efficiency. While the current model processes audio in short, independent frames (512 samples), future work will explore utterance-based processing to capture longer-term dependencies and more explicit phase modeling to further enhance perceptual quality.

Summary updated

BibTeX
@inproceedings{fu2017raw,
  author = {Szu-Wei Fu and Yu Tsao and Xugang Lu and Hisashi Kawai},
  title = {{Raw waveform-based speech enhancement by fully convolutional networks}},
  booktitle = {APSIPA},
  year = {2017},
  doi = {10.1109/apsipa.2017.8281993},
}
Figure