Audio Feature Extraction with Convolutional Neural Autoencoders with Application to Voice Conversion

Golnooshsadat Elhami, Romann M. WeberInfoscience 2019

Abstract

Feature extraction is a key step in many machine learning and signal processing applications. For speech signals in particular, it is important to derive features that contain both the vocal characteristics of the speaker and the content of the speech. In this paper, we introduce a convolutional auto-encoder (CAE) to extract features from speech represented via proposed short-time discrete cosine transform (STDCT). We then introduce a deep neural mapping at the encoding bottleneck to enable converting a source speaker’s speech to a target speaker’s speech while preserving the source-speech content. We further compare this approach to clustering-based and linear mappings.

1. Introduction

Voice conversion (VC) aims to modify a source speaker's speech to sound like a target speaker while preserving the linguistic content. A fundamental challenge in VC is extracting features that effectively disentangle speaker identity from speech content. Traditionally, the Short-Time Fourier Transform (STFT) is used, but its complex-valued nature often leads to the loss of phase information during processing, complicating high-quality reconstruction.

This paper proposes a pipeline using the Short-Time Discrete Cosine Transform (STDCT) combined with a Convolutional Autoencoder (CAE). This approach provides a real-valued representation and achieves significant data compression, facilitating efficient non-linear mapping between speakers.

2. Methodology

2.1 STDCT Feature Extraction

The pipeline begins with the STDCT. Unlike the STFT, the DCT of a real signal is real-valued and exhibits high energy compaction (sparsity). By retaining only the first 256 coefficients, the system achieves an initial 4x compression while maintaining signal integrity.

2.2 Convolutional Autoencoder (CAE) Architecture

The STDCT frames are organized into 2D segments (e.g., 8 consecutive frames) and fed into a CAE. The encoder consists of three convolutional layers designed to capture hierarchical spectral patterns. These layers are followed by a fully connected layer that compresses the representation into a 64-dimensional bottleneck vector. This results in a total compression factor of 128x relative to the original input.

Architecture of the proposed CAE and STDCT/ISTDCT processing blocks.

2.3 Voice Mapping Strategies

To perform conversion, the 64-D latent representation of the source speaker must be mapped to the corresponding representation of the target speaker. Three mapping techniques were evaluated:

  • Linear Mapper: A simple least-squares transformation matrix.
  • Clustering Mapper: A codebook approach using k-means (2000 clusters) to link source and target codewords.
  • Deep Neural Mapper (DNN): A deep network with three hidden layers (512 neurons each) designed to model the non-linear complexities of human speech.

3. Experimental Setup

The models were trained on a parallel corpus from the audiobook Alice’s Adventures in Wonderland, featuring a male source narrator and a female target narrator. The dataset included 2.5 hours for training, 10 minutes for validation, and 5 minutes for testing. Temporal alignment between speakers was achieved using FastDTW (Dynamic Time Warping).

4. Results and Discussion

Performance was evaluated using Mel-cepstral distance (MCD), where lower values indicate higher similarity to the target speaker.

Method MCD
Baseline (Source vs. Target) 26.80
Linear Mapper 26.64
Clustering Mapper 26.58
Deep Neural Mapper (DNN) 26.36

While numerical improvements were modest, the DNN mapper consistently outperformed the linear and clustering baselines. Qualitative analysis of the reconstructed waveforms shows that the DNN mapper better preserves the envelope and rhythmic structure of the target speech compared to simpler mapping methods.

Comparison of source speech waveform and converted waveforms using DNN, Clustering, and Linear mapping methods.

5. Conclusion

The combination of STDCT and CAE provides a robust, real-valued, and highly compressed feature space for voice conversion. The superiority of the DNN mapper highlights the necessity of non-linear modeling in capturing the complexities of speaker identity. Future work will explore non-parallel training data and real-time implementation constraints.

Summary updated

Abstract

Feature extraction is a key step in many machine learning and signal processing applications. For speech signals in particular, it is important to derive features that contain both the vocal characteristics of the speaker and the content of the speech. In this paper, we introduce a convolutional auto-encoder (CAE) to extract features from speech represented via proposed short-time discrete cosine transform (STDCT). We then introduce a deep neural mapping at the encoding bottleneck to enable converting a source speaker’s speech to a target speaker’s speech while preserving the source-speech content. We further compare this approach to clustering-based and linear mappings.
BibTeX
@inproceedings{elhami2019audio,
  author = {Golnooshsadat Elhami and Romann M. Weber},
  title = {{Audio Feature Extraction with Convolutional Neural Autoencoders with Application to Voice Conversion}},
  booktitle = {Infoscience},
  year = {2019},
}
Figure