GPU-Friendly Local Regression for Voice Conversion

Taylor Berg-Kirkpatrick, Dan Klein

Abstract

Voice conversion (VC) requires mapping the complex, non-linear relationships between different speakers' vocal tracts. While non-parametric models like Local Linear Regression (LLR) offer high fidelity, their computational intensity on standard CPUs limits real-time application. This paper presents a "GPU-friendly" LLR implementation that redesigns neighborhood searches to favor contiguous memory access. By sorting training data along the first principal component, the proposed method achieves a 10x speedup over optimized CPU implementations, enabling high-quality voice swapping at nearly 3x real-time speeds.

Introduction

Voice conversion is the process of modifying a source speaker's speech to sound like a target speaker while maintaining the original message. This task involves a classic trade-off between speed and quality. Parametric models, such as Gaussian Mixture Models (GMMs), are computationally efficient but often produce muffled or distorted audio due to over-simplification. Conversely, non-parametric models like LLR provide higher accuracy by calculating unique transformations for every audio frame based on similar training examples. However, LLR's standard "search and skip" neighbor selection creates a massive memory bottleneck on GPUs, as it necessitates slow, random memory accesses.

Methodology

To address the memory bottleneck, the authors propose a surrogate neighborhood function designed specifically for GPU architectures. The process involves three primary steps:

  1. Dimensionality Reduction and Sorting: Principal Component Analysis (PCA) is performed on the training data. All samples are projected onto the first principal component and sorted along this 1D axis.
  2. Contiguous Extraction: For any new input frame, the model identifies neighbors within this 1D projected space. Because the data is pre-sorted, these neighbors are stored in contiguous blocks in memory, allowing the GPU to fetch them at high speed.
  3. Weighted Regression: A weighted least-squares problem is solved using these neighbors. Crucially, while selection happens in 1D, the weighting uses a Gaussian kernel in the original high-dimensional space to ensure accuracy.
PCA-based neighborhood selection for contiguous memory access.

This approach prioritizes memory locality over absolute search precision, operating on the principle that fetching "slightly less perfect" neighbors quickly is more efficient than fetching "perfect" neighbors via random access.

Experimental Setup

The system was evaluated using the CMU Arctic database, a standard benchmark for voice conversion. The training set consisted of 70 sentences from a US female speaker and a Scottish male speaker, with 20 sentences reserved for testing. Speech was parameterized into 24-dimensional mel-cepstral coefficients representing the vocal tract.

Results and Discussion

The GPU-based LLR was compared against a GMM baseline and a multi-threaded CPU LLR implementation running on a 4.4GHz Intel processor.

Comparison of conversion quality and processing speed across different systems.

As shown in the results, the GPU implementation achieved a 10x speedup over the optimized CPU version, processing speech at 2.7x to 2.9x faster than real-time. In contrast, the CPU version operated at only 0.3x real-time. Importantly, this speed was achieved with virtually no loss in quality; the Cepstral Distortion (CD) for GPU LLR (~5.98) was nearly identical to CPU LLR (~5.99) and significantly better than the GMM baseline (~7.0).

Conclusion

By aligning algorithmic search patterns with GPU hardware strengths, the authors demonstrated that non-parametric voice conversion can be both high-fidelity and real-time. While the 1D neighborhood approximation relies on the data having a dominant direction of variance, the hybrid weighting approach maintains robustness. This "contiguous block" strategy for local regression holds potential for other real-time applications in natural language processing and computer vision.

Summary updated

Abstract

Voice conversion is the task of transforming a source speaker’s voice so that it sounds like a target speaker’s voice. We present a GPUfriendly local regression model for voice conversion that is capable of converting speech in real-time and achieves state-of-the-art accuracy on this task. Our model uses a new approximation for computing local regression coefficients that is explicitly designed to preserve memory locality. As a result, our inference procedure is amenable to efficient implementation on the GPU. Our approach is more than 10X faster than a highly optimized CPUbased implementation, and is able to convert speech 2.7X faster than real-time.
BibTeX
@inproceedings{bergkirkpatrick2015gpu,
  author = {Taylor Berg-Kirkpatrick and Dan Klein},
  title = {{GPU-Friendly Local Regression for Voice Conversion}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2015},
  doi = {10.3115/v1/n15-1148},
}

Loading related content...

Figure