TL;DR
This paper introduces a novel adaptive filter-based approach for modifying speech prosody (pitch and duration). By dynamically adjusting an adaptive filter's cutoff frequency based on the estimated maximum frequency of speech segments, the method achieves more accurate pitch tracking, especially for signals with large pitch variations like singing. This leads to improved prosody modification compared to existing techniques.
What Problem Does This Paper Address?
Modifying the prosody of speech—specifically its pitch and duration—is crucial for applications like text-to-speech synthesis, language learning, and artistic vocal processing. Existing methods often struggle with high-quality modifications, particularly for signals with significant pitch variations such as singing. Traditional approaches like filter-based pitch shifting or time-domain methods (e.g., PSOLA) can introduce distortions or require careful manual parameter tuning. The challenge lies in developing a robust and adaptable method that can accurately track pitch and modify prosody naturally across a wide range of speech signals.
Methods
The proposed approach consists of three main components:
- Frequency Estimator: This module first detects sound events (onset and offset) using the CUEX algorithm. It then segments the audio into sentences and estimates the maximum frequency level within each sentence. The YIN algorithm is used for fundamental frequency estimation, and the frequency is converted to a MIDI note number for easier processing.
- Adaptive Filter-Based Pitch Tracker: This is the core innovation. Instead of a fixed filter, an adaptive low-pass elliptic filter is employed. Crucially, the filter's cutoff frequency is dynamically adjusted based on the estimated maximum frequency level of each speech segment. This adaptation allows the filter to better isolate the fundamental harmonic of the speech signal, leading to more accurate pitch estimation, especially for signals with varying fundamental frequencies. The output of this filter is then fed to a zero-crossing detector to estimate the pitch period.
- Prosody Modifier: This module modifies the pitch and duration of the speech signal based on the estimated pitch and user-defined targets. It utilizes a method similar to pitch shifting and time-domain modification techniques, where signal segments are repeated or removed for duration changes, and periods are compressed or stretched for pitch changes. Hanning windows are applied to smooth transitions between modified segments, aiming to preserve naturalness.
Datasets
The paper does not explicitly mention specific training datasets. For evaluation, it states that four sentences spoken by two male and two female speakers were used. These sentences were downloaded from a provided link (http://speech.cs.iitm.ernet.in/Main/result/prosody.html), which is associated with a previous work [2].
Key Insights and Ideas
- Adaptive Filtering for Pitch Tracking: The most significant insight is the use of an adaptive low-pass elliptic filter whose cutoff frequency is tuned to the estimated maximum frequency of each speech segment. This allows the filter to "follow" the signal's frequency characteristics, leading to more robust pitch estimation than fixed-filter approaches.
- Bridging Singing and Speech: The adaptive nature of the filter is highlighted as particularly beneficial for signals with large pitch variations, such as singing, where fixed-parameter methods often fail.
- Improved Naturalness through Accurate Pitch: By achieving more accurate pitch tracking, the subsequent prosody modification is less prone to introducing artifacts and distortions, leading to a more natural-sounding output.
Key Results
The paper presents subjective evaluation results comparing the proposed adaptive filter-based approach with two other methods: LP-PSOLA and an epoch-based approach.
- Pitch Modification: For pitch modification factors of 1.33, 0.66, 0.5, and 0.4, the proposed approach consistently received higher mean opinion scores (MOS) than both LP-PSOLA and the epoch-based method, indicating better perceived quality and naturalness. For a pitch modification factor of 2, it was slightly inferior to LP-PSOLA but still better than the epoch-based method.
- Duration Modification: For duration modification factors of 0.5, 0.75, 1.5, and 2, the proposed adaptive filter-based approach again outperformed both LP-PSOLA and the epoch-based approach across all tested factors.
Limitations and Future Work
- Computational Cost: The authors acknowledge that the frequency estimator, particularly the YIN algorithm, is the most computationally intensive part of the system. However, they suggest that with reasonable audio signal sample rates and frame sizes, real-time implementation is feasible.
- Naturalness Degradation: Despite improvements, some degradation in the naturalness of synthesized speech is still observed, especially when pitch or duration modification factors are very large or very small. This is attributed to issues like excessive silence between periods (for large factors) or signal squeezing (for small factors), and the deletion/replication of period signals leading to unnatural connections.
- Future Work: The authors suggest further research into resolving the naturalness degradation issues, particularly concerning the connection of modified signal segments. They also mention that more sophisticated algorithms could be developed to optimize parameters like
addNote.
Summary updated