TL;DR
This paper tackles the challenge of making synthetic Czech speech sound more natural and expressive by developing a more nuanced way to represent different "communicative functions" (like asking a question or expressing thanks). Instead of treating all differences equally, they created a "penalty matrix" that quantifies how similar or different these functions are, leading to improved expressivity in synthesized speech.
What Problem Does This Paper Address?
Current text-to-speech (TTS) systems, while capable of producing high-quality speech, often lack the expressivity needed for natural human-computer interaction. To make synthetic voices more engaging, especially in limited-domain applications like virtual companions, researchers are exploring ways to imbue them with expressivity. A key challenge is how to represent and synthesize these expressive nuances. The authors focus on "communicative functions" (CFs) as a way to categorize expressivity, building on previous work that used a simple penalty system in unit selection TTS. This system penalized any mismatch between a target CF and a candidate speech unit equally. The problem is that not all CFs are equally distinct; for example, "asking a question" and "expressing thanks" are acoustically and perceptually different, and this difference should be reflected in the synthesis process. The paper aims to move beyond a binary penalty system to a more graded approach that captures these subtle differences.
Methods
The core of this paper is the development of a penalty matrix that quantifies the differences between various communicative functions (CFs). This matrix is crucial for the unit selection TTS algorithm, where it influences the selection of speech units based on their CF.
The authors employed a two-pronged approach to build this matrix:
- Listening Test Based Differences: They used annotations from a listening test where human listeners labeled expressive speech recordings with CFs. The penalty (dissimilarity) between two CFs was calculated based on how often recordings labeled with one CF were perceived as belonging to another. This was done using a logarithmic scale to better reflect human perception.
- Acoustic Analysis Based Differences: They analyzed acoustic features (F0, RMS energy, and unit duration) of speech units associated with each CF. The penalty between CFs was then calculated as the Euclidean distance between the average acoustic feature vectors for those CFs.
Finally, these two matrices were combined into a single final penalty matrix. The authors weighted the listening test-based differences more heavily, believing human perception is paramount for expressivity. The formula for combining them is: $A_{ij} = \frac{3 \times a_{ij} + a'_{ij}}{K}$, where $a_{ij}$ is from the listening test, $a'_{ij}$ is from acoustic analysis, and $K$ is a constant (set to 6 in their experiments).
Datasets
The paper utilizes an expressive speech corpus of Czech conversations between seniors and a computer, focusing on personal photographs as a topic. This corpus was collected and annotated with communicative functions. Specific details about the corpus size or exact number of speakers are not provided beyond its purpose.
Key Insights and Ideas
- Expressivity is Graded, Not Binary: The most significant insight is that communicative functions are not all equally distinct. Treating all mismatches as equal in a TTS system leads to suboptimal expressivity. A nuanced penalty system that reflects perceptual and acoustic differences is necessary.
- Human Perception is Key: The authors emphasize that human perception of expressivity is more critical than purely acoustic differences when designing a penalty matrix for TTS. This led them to give more weight to the listening test results.
- Limited Domain Benefits from Nuance: While expressive speech synthesis is generally complex, focusing on a limited domain (like senior-computer interactions) makes it more tractable to define and implement specific communicative functions and their differences.
- Trade-offs Exist: Implementing a more complex penalty matrix can improve expressivity but might also affect speech quality or smoothness of concatenations, necessitating careful evaluation.
Key Results
- Improved CF Matching: The new penalty matrix led to a significant increase in the selection of speech units with the appropriate communicative function. This was observed as a 109% improvement for neutral texts and a 9% improvement for expressive texts when comparing the proportion of correctly matched CF units.
- Slight Smoothness Deterioration: The new system showed a noticeable decrease in the "smoothness" of concatenated speech units for neutral texts (from 65% to 52% natural concatenation points). However, this deterioration was almost imperceptible for expressive texts.
- MOS Score Remains Stable: A listening test using the Mean Opinion Score (MOS) revealed a slight, statistically insignificant decrease in perceived speech quality (from 3.5 to 3.4) for the experimental system compared to the baseline. This suggests that while smoothness might be slightly impacted for neutral content, overall quality remains acceptable.
Limitations and Future Work
- Limitations:
- The authors acknowledge that the set of communicative functions is specific to their limited domain and not a general solution.
- The constant $K$ in the penalty matrix calculation was set "ad-hoc" for preliminary experiments, suggesting it could be further optimized.
- The evaluation of speech quality was limited to expressive text content for practical reasons.
- The study focused on a single speaker for acoustic analysis.
- Future Work:
- The authors plan to conduct another listening test specifically to assess the level of expressivity as perceived by humans, which is a more complex evaluation than the current MOS test.
- Further investigation into feature weighting within the target cost function is suggested as a way to improve CF matching.
Summary updated