Learning speaker normalization using semisupervised manifold alignment

Andrew R. Plummer, Mary E. Beckman, Mikhail Belkin, Eric Fosler-Lussier, Benjamin MunsonInterspeech 2010 · DOI · OpenReview

Introduction

A fundamental challenge in language acquisition is speaker normalization. When an infant hears an adult produce a vowel, the acoustic properties—specifically the formant frequencies—differ significantly from the infant's own productions due to differences in vocal tract size. For instance, an infant's resonance frequencies are much higher than an adult's. To learn language, infants must recognize that these disparate acoustic signals represent the same linguistic units. This paper proposes that this normalization is not innate but a learned process modeled through semisupervised manifold alignment.

Methodology: Manifold Alignment

The researchers treat vowel spaces as low-dimensional manifolds (surfaces) embedded within a higher-dimensional space of formant frequencies (F1-F4). The normalization process is divided into two tasks:

  1. Grouping: Clustering similar sounds within a single speaker's acoustic space (e.g., identifying all instances of /i/).
  2. Mapping: Establishing correspondences between the adult's clusters and the infant's clusters.

The core of the approach is Semisupervised Manifold Alignment. By using a small set of "exemplar pairs" (known associations between an adult sound and an infant sound), the algorithm preserves the local geometry of each speaker's vowel space while projecting both into a shared intermediate space where they overlap.

Visualization of the vowel spaces for an adult and infant (a, b) and the resulting alignment in a shared manifold space (c).

Experimental Setup

The study utilized synthetic acoustic data from the Variable Linear Articulatory Model (VLAM), simulating a 6-month-old male and a 16-year-old female. Formant values were converted to the Bark scale to better align with human auditory perception. To evaluate the model's cross-linguistic applicability, the researchers used perceptual labels from native speakers of Greek, Korean, and English, allowing them to test the model against different phonological systems.

Results and Discussion

The model demonstrated high proficiency in the "Grouping" task, with accuracy ranging from 75% to 93%. The "Mapping" task proved more complex but yielded promising results, particularly for English.

Mapping accuracy percentages for different languages based on the number of eigenvectors and formants used.

As shown in the table above, English mapping accuracy reached 83% when using four formants and two eigenvectors. Greek and Korean accuracies were generally lower, typically hovering between 50% and 67%. The authors suggest that the density and inventory of a language's vowel space significantly impact the difficulty of the alignment task. Despite these variations, the results provide a computational proof-of-concept that the geometric structure of vowel spaces provides sufficient information for a learning-based normalization process.

Conclusion

While the model currently relies on synthetic data and assumes the existence of initial exemplar pairs, it offers a viable framework for understanding how humans bridge the acoustic gap between speakers. Future research will focus on transitioning to natural speech data and investigating how infants might autonomously acquire the initial "anchor" associations needed to initiate the alignment process.

Summary updated

Abstract

As a child acquires language, he or she: perceives acoustic information in his or her surrounding environment; identifies portions of the ambient acoustic information as language-related; and associates that language-related information with his or her perception of his or her own language-related acoustic productions. The present work models the third task. We use a semisupervised alignment algorithm based on manifold learning. We discuss the concepts behind this approach, and the application of the algorithm to this task. We present experimental evidence indicating the usefulness of manifold alignment in learning speaker normalization.
BibTeX
@inproceedings{plummer2010learning,
  author = {Andrew R. Plummer and Mary E. Beckman and Mikhail Belkin and Eric Fosler-Lussier and Benjamin Munson},
  title = {{Learning speaker normalization using semisupervised manifold alignment}},
  booktitle = {Interspeech},
  year = {2010},
  doi = {10.21437/interspeech.2010-758},
}
Figure