Frequency-domain delexicalization using surrogate vowels

Alexander Kain, Jan P. H. van SantenInterspeech 2010

TL;DR

This paper introduces a novel frequency-domain delexicalization technique that effectively removes lexical content from speech while preserving crucial prosodic cues like naturalness and speaker identity. This method is particularly useful for studies investigating prosody in contexts where word meaning could be a confounding factor, such as research on affective prosody in individuals with autism.

What Problem Does This Paper Address?

Delexicalization, the process of rendering speech unintelligible while retaining its prosodic characteristics, is essential for various research applications, especially in studying prosody. Traditional delexicalization methods often strip away important acoustic features beyond fundamental frequency (F0), duration, and energy. These lost features, such as spectral tilt and voice quality, are critical for conveying nuanced prosodic information like word emphasis and affective states. The challenge is to create a delexicalized signal that is not only unintelligible but also preserves these subtle prosodic cues to enable accurate perception of affect and maintain naturalness and speaker identity, which are often compromised by existing methods.

Methods

The proposed method operates in the frequency domain and involves two main stages: training and delexicalization.

Training: 1. Sinusoidal Analysis: The input speech is analyzed using a pitch-synchronous, harmonic sinusoidal analysis. This involves identifying glottal closure instances (GCIs) and auxiliary markers to segment the speech into frames. 2. Frame Modeling: Each frame is modeled using sinusoidal parameters (magnitude and phase) based on an asymmetric triangular window. 3. Surrogate Vowel Selection: The user graphically selects a spectral slice from a "surrogate vowel" within a training utterance. The sinusoidal parameters and F0 of this selected frame are stored.

Delexicalization: 1. Analysis: The input utterance is analyzed identically to the training phase. 2. Voiced Frames: For voiced frames, the magnitude parameters within a specific frequency band (200 Hz to 7000 Hz) are replaced by those from the selected surrogate vowel. The energy in this band is adjusted to match the original, while phase information is preserved. 3. Unvoiced Frames: For unvoiced frames, the magnitude parameters in a different frequency band (400 Hz to 6000 Hz) are replaced by the average energy in that band.

This approach aims to preserve spectral detail, short-term and long-term energy fluctuations, micro and macro intonation, and the impulsive/fricative nature of consonants by selectively modifying magnitude while retaining phase and F0.

Datasets

The paper does not explicitly list or introduce new datasets for training or evaluation. Instead, it uses:

  • Training Utterances: User-selected utterances for capturing surrogate vowel spectral slices.
  • Evaluation Stimuli: Conversationally spoken sentences (semantically anomalous) and sentences designed to elicit specific affects (angry, fearful, happy, sad). These were derived from or inspired by existing resources like the IEEE Harvard Psychacoustic Sentences and vignettes for affect elicitation.

Key Insights and Ideas

  • Frequency-domain manipulation is key: By operating on spectral slices and averaging energy, the method can selectively remove lexical information while retaining broader spectral characteristics that contribute to prosody.
  • Preserving phase is crucial for naturalness: The decision to keep phase information unmodified is hypothesized to help retain the impulsive or fricative nature of consonants and contribute to overall signal quality.
  • Surrogate vowels as building blocks: Using spectral slices from a user-selected vowel provides a consistent, albeit artificial, spectral template for voiced segments, aiming to maintain a semblance of vocal tract resonance without specific phoneme identification.
  • Affective prosody is complex: The results suggest that while delexicalization can preserve some affective cues, it's not a perfect process, and the "disconnect" between lexical and prosodic content in certain populations (like individuals with autism) is a complex area requiring sophisticated analysis.

Key Results

  • Intelligibility: Near-complete unintelligibility was achieved for both delexicalized conditions. Keyword recall was 0.8% for the "delex-classic" baseline and 0.0% for the "delex-proposed" method, compared to 91.6% for the original speech.
  • Affect Recognition: Affect recognition rates were 45.8% for "delex-classic" and 49.0% for "delex-proposed," both significantly lower than the 65.6% for the original speech. The difference between the two delexicalized methods was not statistically significant.
  • Naturalness and Speaker Identity: In preference tests, 64% of listeners preferred the "delex-proposed" method for naturalness, citing the "delex-classic" as "too muffled." Crucially, 88% preferred the "delex-proposed" method for preserving speaker identity, a significant improvement over the baseline.

Limitations and Future Work

  • Manual Surrogate Vowel Selection: The current implementation requires manual selection of the surrogate vowel slice. The authors state that work is underway to automate this step.
  • Affect Recognition Improvement: While the method preserves more speaker identity and naturalness, the improvement in affect recognition over the baseline was modest.
  • Future Directions:
    • Automating the training process (surrogate vowel selection).
  • Investigating how spectral balance can be further preserved during delexicalization.
  • Exploring the use of speaker-adapted Hidden Markov Models to recognize broad phonetic classes, which could then be replaced by associated surrogate phonemes for potentially improved naturalness.

Summary updated

BibTeX
@inproceedings{kain2010frequency,
  author = {Alexander Kain and Jan P. H. van Santen},
  title = {{Frequency-domain delexicalization using surrogate vowels}},
  booktitle = {Interspeech},
  year = {2010},
}
Figure