Abstract
We present a multimodal open-set speaker identification system that integrates information coming from audio, face and lip motion modalities. For fusion of multiple modalities, we propose a new adaptive cascade rule that favors reliable modality combinations through a cascade of classifiers. The order of the classifiers in the cascade is adaptively determined based on the reliability of each modality combination. A novel reliability measure, that genuinely fits to the open-set speaker identification problem, is also proposed to assess accept or reject decisions of a classifier. The proposed adaptive rule is more robust in the presence of unreliable modalities, and outperforms the hard-level max rule and soft-level weighted summation rule, provided that the employed reliability measure is effective in assessment of classifier decisions. Experimental results that support this assertion are provided.
Introduction
Speaker identification systems operating in real-world environments face significant challenges. Audio-only systems are highly susceptible to background noise, while visual modalities like face and lip motion are sensitive to lighting and pose variations. To mitigate these issues, multimodal fusion is typically employed. However, standard fusion techniques often struggle with two key aspects: the "open-set" problem (where the system must correctly reject impostors) and the dynamic nature of modality reliability (e.g., audio becoming less reliable as noise increases).
Adaptive Classifier Cascade
This paper introduces an adaptive classifier cascade designed to dynamically prioritize the most reliable information. Instead of a static fusion of all scores, the system employs a hierarchical decision process:
- Dynamic Ordering: Classifiers and their combinations are ranked based on a novel reliability measure ($R_p$).
- The Stop Rule: The system evaluates the most reliable classifier first. If its confidence—measured by the "likelihood offset" between the top candidate and the rejection threshold—is sufficiently high, a final decision is made immediately.
- Escalation: If the initial classifier is uncertain, the system moves to the next level of the cascade, incorporating additional modality combinations. This prevents noisy data (like audio in a loud room) from corrupting a clear decision made by a more reliable modality (like a clear face image).
A Novel Reliability Measure ($R_p$)
Effective cascading requires an accurate assessment of classifier confidence. The proposed $R_p$ measure is specifically designed for open-set identification. It balances Accept Evidence (high likelihood for the top candidate and a significant gap from the runner-up) with Reject Evidence (low overall likelihood and high confusion between candidates), allowing the system to handle both known users and impostors more effectively.
Experimental Results
The system was evaluated using the MVGL-AVD audio-visual database, featuring 50 subjects and varying levels of "office and babble" noise. Performance was measured using the Equal Error Rate (EER), where lower values indicate better performance.
As shown in the results table, the adaptive cascade significantly outperforms individual modalities and standard fusion rules, especially in high-noise environments. For instance, at a noisy 5dB SNR, audio-only identification (A) suffers from a 26.5% EER. The proposed adaptive cascade configuration ($M_0 \cdot A \cdot F$) reduces this to 4.5% EER, and the most complex configuration ($M_0 \cdot M_1 \cdot M_2 \cdot A \cdot F$) reaches as low as 2.3% EER.
Furthermore, the adaptive approach demonstrates superior robustness compared to standard fusion. At 10dB SNR, a common fusion method ($A \oplus F$) yields a 3.2% EER, whereas the cascade configuration $M_0 \cdot M_1 \cdot A \cdot F$ achieves a much lower 1.1% EER.
Conclusion
The adaptive classifier cascade offers a robust framework for multimodal speaker identification. By gating the fusion process through reliability-based escalation, the system avoids the pitfalls of static weighting. While the current implementation shows substantial gains, future work will focus on reducing computational complexity and automating the tuning of decision thresholds to make the system more practical for real-time applications.
Summary updated