Abstract
Unsupervised speech pattern discovery aims to identify recurring segments in speech without prior knowledge of the language or a pre-trained recognizer. This paper explores the integration of visual information—specifically lip movements—to enhance the robustness of this process. By utilizing Canonical Correlation Analysis (CCA) to synchronize asynchronous audio and visual streams and implementing a decision fusion approach, the researchers achieved a 10.05% relative improvement in F1-score over traditional audio-only methods.
Introduction
The task of unsupervised speech pattern discovery is akin to learning a language from scratch by listening to a radio broadcast. While certain acoustic patterns repeat, identifying word boundaries and distinguishing between phonetically similar terms (e.g., "white" vs. "five") remains a significant challenge for audio-only systems. This research addresses these ambiguities by incorporating visual cues from the speaker's mouth, providing a secondary, independent modality to verify acoustic matches.
Methodology
The proposed framework relies on three core components: pattern searching, temporal synchronization, and multimodal fusion.
Unbounded Dynamic Time Warping (UDTW)
UDTW serves as the primary search mechanism. It compares pairs of utterances to find paths of high similarity, effectively identifying where specific patterns repeat across different segments of speech.
Synchronization via Canonical Correlation Analysis (CCA)
A critical challenge in audiovisual processing is the inherent asynchrony between lip movements and sound production; the mouth often begins moving before any sound is emitted. To resolve this, the authors use CCA to find the optimal temporal offset that maximizes the correlation between audio and video frames.
Fusion Strategies
The study evaluated three distinct fusion methods:
- Audio-Visual Feature Concatenation (AVFC): Merging audio and visual feature vectors into a single input.
- Audio-Visual Similarity Weighting (AVSW): Averaging similarity scores calculated independently for each modality.
- Audio-Visual Decision Fusion (AVDF): Running discovery independently on both streams and retaining only those patterns identified by both. This "consensus" approach proved most effective at filtering out false positives.
Experimental Setup
The system was evaluated using the GRID audiovisual corpus, focusing on 696 videos from a single speaker. Audio was represented using Mel-frequency cepstral coefficients (MFCCs), while visual features were extracted from lip coordinates and "eigenmouth" parameters derived via Principal Component Analysis (PCA).
Results and Discussion
The performance was measured using the F1-score, which balances precision and recall.
| Modality | F1-Score |
|---|---|
| Audio-Only (AO) | 53.11% |
| Visual-Only (VO) | 35.76% |
| AVDF + CCA (Proposed) | 58.45% |
The proposed AVDF+CCA method yielded a 10.05% relative improvement over the audio-only baseline. The primary benefit of the visual modality was its ability to disambiguate acoustically similar words. For instance, while "white" and "five" may sound similar to an unsupervised algorithm, the distinct lip movements for "w" and "f" allow the system to correctly reject false acoustic matches.
Conclusion
This research demonstrates that visual information significantly enhances unsupervised speech pattern discovery when properly synchronized and fused. While currently limited to a single-speaker, controlled environment, the results provide a strong foundation for future work on multi-speaker datasets and more complex, "in-the-wild" acoustic conditions.
Summary updated