Adaptive articulatory feature-based conditional pronunciation modeling for speaker verification

Kwong-Sak Leung, Man‐Wai Mak, M.h. Siu, Sun‐Yuan KungSpeech 2005 · DOI

TL;DR

This paper introduces a novel approach to speaker verification by modeling how speakers pronounce sounds, not just the sounds themselves. By linking articulatory features (like tongue and lip movements) to phonemes, they create "conditional pronunciation models" that capture unique speaking styles. Fusing these models with traditional spectral analysis significantly improves accuracy, especially in challenging conditions like varying phone quality.

What Problem Does This Paper Address?

Traditional speaker verification systems primarily rely on analyzing the acoustic properties of speech, like the frequency content of sounds. While effective, these methods often overlook the subtle, yet distinctive, ways individuals articulate speech. Differences in accent, education, and even personal habits lead to unique "pronunciation styles" that carry valuable speaker-specific information. Existing methods that try to capture this, like Conditional Pronunciation Modeling (CPM), often require vast amounts of multilingual data and long speech samples, making them impractical for many real-world scenarios. This paper aims to bridge this gap by developing a more efficient and effective CPM system that leverages articulatory features, which are language-independent and can be extracted from shorter speech segments.

Methods

The core innovation lies in the Articulatory Feature-Based Conditional Pronunciation Modeling (AFCPM). Here's how it works:

  • Articulatory Feature (AF) Extraction: Instead of directly analyzing acoustic signals, the system first extracts articulatory features. These features represent abstract classes of tongue and lip movements (e.g., "manner" like stop, fricative, and "place" like labial, dental). This is achieved using two Multilayer Perceptrons (MLPs) trained to map spectral features to probabilities of these articulatory classes.
  • Phoneme Alignment: A standard phoneme recognizer is used to obtain a sequence of phonemes from the speech signal.
  • Conditional Pronunciation Modeling (CPM): The AF streams and the phoneme sequence are then aligned. For each speaker, probabilistic models are built that capture the likelihood of specific articulatory feature combinations given a particular phoneme. This essentially models how a speaker articulates a given sound.
  • Fusion with Spectral Features: The AFCPM scores are then fused with scores from a traditional spectral-based system (using MFCCs). Crucially, a frame-dependent weighting scheme is introduced. This scheme uses the confidence of the articulatory feature extraction (specifically, the probability of the "manner" class) to dynamically adjust the contribution of each frame's score to the overall utterance score. This allows the system to give more weight to frames where the articulatory information is more reliable.

Datasets

  • HTIMIT corpus: Used for training the Articulatory Feature (AF) MLPs. This corpus is derived from TIMIT, with added transducer distortions to simulate real-world phone quality variations.
  • SPIDRE corpus: Used for evaluating the AFCPM system. This corpus is a subset of Switchboard, containing conversations from target and non-target speakers.

Key Insights and Ideas

  • Articulatory Features are Speaker-Discriminative: The paper strongly suggests that the way a speaker moves their articulators to produce sounds carries significant, unique information for speaker identification.
  • Language Independence: AFs are inherently language-independent, meaning the training data requirements are significantly reduced compared to previous CPM methods that relied on multilingual phone models.
  • Efficiency Gains: AFCPM allows for effective speaker verification with much shorter utterances for both enrollment and testing, making it more practical.
  • Confidence-Based Fusion is Key: Dynamically weighting the fusion of spectral and articulatory features based on the confidence of the articulatory information significantly boosts performance, especially when acoustic conditions are poor.
  • Complementary Information: Articulatory features provide information that is complementary to spectral features, leading to improved robustness, particularly in mismatched conditions (e.g., different microphones).

Key Results

The paper reports significant improvements in speaker verification performance:

  • AFCPM System Performance: The AFCPM system alone achieved a lower Equal Error Rate (EER) than the baseline MFCC system.
  • Fusion Performance:
    • When using recognized phoneme alignments, the fused system achieved an EER of 14.44%, representing an 8.83% error reduction compared to the MFCC baseline.
  • With forced phoneme alignments (assuming perfect phoneme recognition), the fused system achieved an EER of 13.26%, representing a 16.28% error reduction.
  • Robustness: The fusion approach showed particular effectiveness in reducing errors under mismatched handset conditions, where spectral features are less reliable.

Limitations and Future Work

  • Phoneme Alignment Accuracy: The results highlight that the accuracy of phoneme alignment is critical for the AFCPM system's performance. Errors in alignment can negatively impact the verification score.
  • Data Requirements: While significantly reduced compared to prior CPM methods, the system still requires some amount of training data for speaker enrollment.
  • Future Work: The authors suggest that further exploration of different articulatory feature representations and more sophisticated fusion techniques could lead to even greater performance gains. Investigating the impact of different weighting schemes and exploring the use of AFs in other speech processing tasks are also potential avenues.

Summary updated

BibTeX
@inproceedings{leung2005adaptive,
  author = {Kwong-Sak Leung and Man‐Wai Mak and M.H. Siu and Sun‐Yuan Kung},
  title = {{Adaptive articulatory feature-based conditional pronunciation modeling for speaker verification}},
  booktitle = {Speech},
  year = {2005},
  doi = {10.1016/j.specom.2005.05.013},
}
Figure