Abstract
This paper presents a novel self-supervised framework for the acquisition of American English vowel categories. By leveraging the temporal correlation between auditory speech (formants) and visual lip movements, the system discovers phonetic structures without human-labeled data or prior linguistic knowledge. The approach utilizes cross-modal clustering and Hebbian projections to ground perceptual categories, successfully identifying ten distinct vowel monophthongs from raw sensory input.
Introduction
The challenge of perceptual grounding—how an agent identifies meaningful categories from raw sensory data—is central to both cognitive science and artificial intelligence. In speech processing, identifying vowels is particularly difficult due to the overlapping and ambiguous nature of acoustic formants. While traditional machine learning relies on supervised labels, human infants learn these categories through multi-modal exposure. This research proposes that sensory systems can "bootstrap" off one another, using the distinct perspectives of sight and sound to resolve ambiguities and organize perceptions into categories.
Methods
The proposed framework centers on Cross-Modal Clustering. Sensory inputs, such as audio formants and lip shapes, are represented as topological manifolds discretized into Voronoi regions. The system tracks co-occurrences between these modalities using Hebbian projections, which represent the spatial probability distribution of one sense given another.
To compare these distributions, the author introduces a Similarity Distance ($D_S$) based on the Kantorovich-Wasserstein distance (Earth Mover's Distance). This metric allows the system to determine if two distinct sounds belong to the same category if they consistently co-occur with the same visual lip configuration. The study utilized a multi-modal dataset of an adult female speaker reciting ten American English vowels in a "[h]-vowel-[d]" context, extracting auditory formants via Linear Predictive Coding and visual lip axes via a custom tracking system.
Results and Key Insights
The system successfully acquired the structure of ten American English vowels without prior phonetic knowledge. This represents a significant milestone as a rare example of unsupervised phonetic acquisition outside of human biological systems.
The resulting clusters closely align with the established "vowel space" used in linguistics. By sharing information across modalities, the system achieves mutual disambiguation, where visual cues help resolve acoustic overlaps (e.g., distinguishing /i/ from /ɪ/). Notably, the framework is non-parametric, discovering the correct number of clusters (10) automatically rather than requiring a pre-specified value.
Conclusion
This work demonstrates the power of cross-modal correlation for self-supervised learning. While currently limited to a single speaker and monophthongs, the amodal nature of the mathematical framework suggests broad applicability. Future research will focus on scaling the system to handle speaker variability, temporal offsets in real-world signals, and the complexities of continuous natural speech, including diphthongs and consonants.
Summary updated