Mel-spectrogram augmentation for sequence to sequence voice conversion

Yeongtae Hwang, Hyemin Cho, Hongsun Yang, Insoo Oh, Seong-Whan LeearXiv 2020 · OpenReview

TL;DR

This paper tackles the challenge of limited paired data for training sequence-to-sequence voice conversion (VC) models by exploring Mel-spectrogram augmentation. They introduce novel augmentation techniques and a cost-effective hyperparameter search strategy, finding that time-axis warping methods significantly improve VC performance.

What Problem Does This Paper Address?

Training effective voice conversion (VC) models, especially those using sequence-to-sequence architectures, often requires a large number of parallel speech data – recordings of the same utterance spoken by different speakers. Acquiring such paired data is expensive and time-consuming. This scarcity of data can hinder model performance, leading to less natural or accurate voice conversions. The paper aims to address this data bottleneck by investigating how Mel-spectrogram augmentation can artificially increase the diversity of training data, thereby improving the robustness and quality of VC models trained from scratch.

Methods

The core of this paper lies in its exploration and proposal of Mel-spectrogram augmentation techniques. The authors adopt and adapt policies from SpecAugment, a successful augmentation method for Automatic Speech Recognition (ASR), including time masking, frequency masking, and time warping. Crucially, they introduce new augmentation policies:

  • Frequency Warping (FW): Adjusts the pitch by warping frequency points.
  • Loudness Control (LC): Modifies the overall loudness of the speech segment.
  • Time Length Control (TLC): Alters the speech speed by controlling the time length.

To efficiently find the best parameters for these augmentation techniques without the computationally expensive process of retraining the VC model repeatedly, they propose a novel hyperparameter search strategy. This strategy relies on a new metric called the Deformation per Deteriorating (DPD) ratio. The DPD ratio aims to quantify how much a given augmentation parameter deforms the Mel-spectrogram while minimizing the degradation in speech quality (measured by Character Error Rate - CER).

Datasets

  • KSS Dataset: Used as the source speaker dataset, containing 12,853 Korean utterances from a female speaker (approx. 12+ hours).
  • Internal Dataset: An internal dataset from Netmarble was used for the target speaker.
  • Paired Dataset: A paired dataset was constructed by aligning transcripts from the KSS dataset, resulting in 12,798 utterances (approx. 8+ hours each).
  • Validation and Test Sets: 64 utterances were used for validation and 64 for testing. The remaining data was used for training.

Key Insights and Ideas

  • Time-Axis Warping is King: The most significant finding is that augmentation policies manipulating the time axis (Time Warping and Time Length Control) consistently outperform other methods. This suggests that variations in speech timing are more crucial for improving VC than frequency-based distortions or loudness changes.
  • Novel Augmentations Offer Value: The proposed frequency warping, loudness control, and time length control provide additional ways to diversify the training data, complementing existing techniques.
  • DPD Metric for Efficient Tuning: The DPD ratio offers a practical solution to the hyperparameter tuning problem. It allows researchers to select optimal augmentation parameters without the prohibitive cost of full model retraining, making the augmentation process much more accessible.
  • Single Policy Dominance: Combining multiple augmentation policies did not yield better results and sometimes led to performance degradation. This implies that over-augmenting can strip away too much linguistic and acoustic information, hindering conversion quality.

Key Results

  • Time Length Control (TLC) and Time Warping (TW) showed the best performance among single augmentation policies when applied to the 1/16 training set, outperforming the baseline.
  • Specifically, 'TLC both' (applying Time Length Control to both source and target audio) was identified as a particularly effective strategy, likely because it creates a one-to-one mapping between source and target utterances.
  • The DPD metric was used to select optimal hyperparameters for each augmentation policy, with the best parameters highlighted in tables. For instance, in Table 2, the maximum DPD values for Time Masking and Frequency Masking were achieved with specific parameter combinations.
  • While not explicitly quantified with a single number, the paper demonstrates through tables (e.g., Table 5, 6, 7) that augmentation, particularly time-axis based, generally leads to improved metrics like lower MCD (Mel-Cepstral Distortion) and CER, and higher naturalness and similarity scores, especially when data is scarce.

Limitations and Future Work

  • Limited Policy Combinations: The paper explored combinations of only a few policies that showed promise individually. The authors suggest that future work should investigate if combining policies, guided by the DPD metric, can lead to further improvements.
  • Over-Augmentation Risk: The study observed that combining multiple augmentation policies could lead to a loss of linguistic and acoustic information, negatively impacting performance. Future research could focus on finding optimal combinations or more sophisticated multi-policy strategies.
  • Hyperparameter Search for Multiple Policies: While the DPD metric is effective for single policies, the paper acknowledges the need to explore its application for finding optimal hyperparameters when combining multiple policies.
  • Focus on Seq2Seq VC: The study specifically targets sequence-to-sequence VC. Further research could explore the applicability and effectiveness of these augmentation techniques on other VC architectures.

Summary updated

Abstract

For training the sequence-to-sequence voice conversion model, we need to handle an issue of insufficient data about the number of speech pairs which consist of the same utterance. This study experimentally investigated the effects of Mel-spectrogram augmentation on training the sequence-to-sequence voice conversion (VC) model from scratch. For Mel-spectrogram augmentation, we adopted the policies proposed in SpecAugment. In addition, we proposed new policies (i.e., frequency warping, loudness and time length control) for more data variations. Moreover, to find the appropriate hyperparameters of augmentation policies without training the VC model, we proposed hyperparameter search strategy and the new metric for reducing experimental cost, namely deformation per deteriorating ratio. We compared the effect of these Mel-spectrogram augmentation methods based on various sizes of training set and augmentation policies. In the experimental results, the time axis warping based policies (i.e., time length control and time warping.) showed better performance than other policies. These results indicate that the use of the Mel-spectrogram augmentation is more beneficial for training the VC model.
BibTeX
@misc{hwang2020mel,
  author = {Yeongtae Hwang and Hyemin Cho and Hongsun Yang and Insoo Oh and Seong-Whan Lee},
  title = {{Mel-spectrogram augmentation for sequence to sequence voice conversion}},
  year = {2020},
  eprint = {2001.01401},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2001.01401},
}
Figure