Abstract
This paper introduces HIT-AVDB-II, a multi-view audio-visual database designed to advance biometric research by capturing "extreme features"—unique habitual mouth movements—across four simultaneous camera angles and varied environmental conditions. Developed at the Harbin Institute of Technology, the database provides a robust benchmark for identifying individuals based on their unique speaking habits, moving beyond the constrained, frontal-only datasets common in the field.
Introduction
While audio-visual speech recognition and biometrics have progressed significantly, real-world application remains a challenge. Most existing databases are recorded under highly constrained conditions: subjects look directly at a single camera in perfect lighting with a limited vocabulary. In practical scenarios, head movement, lighting changes, and idiosyncratic speaking habits significantly impact system performance. HIT-AVDB-II addresses these gaps by focusing on "extreme features" (exaggerated or idiosyncratic mouth movements) which serve as a form of "visual fingerprint" for biometric identification.
The HIT-AVDB-II Database
The HIT-AVDB-II dataset consists of 30 subjects (15 male, 15 female) recorded over multiple sessions. To simulate real-world variability, recordings were conducted in three sessions (morning, afternoon, evening) against three distinct backgrounds: dark blue, light green, and dark red.
The corpus is significantly more diverse than standard digit-based datasets, including:
- Chinese and English poems (to capture emotion and prosody).
- Tongue twisters (to expose fast movements and pronunciation defects).
- The Greek alphabet and musical notes.
Recording Methodology
A sophisticated recording rig was employed to capture synchronized multi-view data. Four cameras recorded subjects simultaneously at 0° (frontal), 30°, 60°, and 90° (profile) angles. This setup allows researchers to investigate how different views complement each other, particularly the profile view, which often contains information missing from frontal perspectives.
The Concept of "Extreme Features"
A core contribution of this work is the emphasis on "extreme features" (ext-features). The authors argue that personal habits—how individuals move their mouths uniquely or exaggerate certain sounds—are highly distinguishing characteristics. By using complex speech tasks like tongue twisters, the database intentionally triggers these idiosyncratic behaviors, providing a richer source of data for biometric systems than "proper" or neutral speech.
Experimental Protocols and Baseline
To ensure fair comparison and prevent "cherry-picking" of results, the authors established two standardized experimental protocols:
- Protocol I (Multi-view): Evaluates algorithm performance on individual views versus various combinations of views.
- Protocol II (Intersection): Tests session-independence by training on one day and testing on another, ensuring the system is robust to changes in lighting or subject appearance.
As a baseline, the researchers implemented a system using 2-D Discrete Cosine Transform (DCT) for feature extraction on 64x64 mouth regions, paired with a Semi-Continuous Hidden Markov Model (SCHMM) for recognition. The system utilizes the top 30 low-frequency DCT components and an SCHMM with 6 states and 8 mixtures.
Conclusion
HIT-AVDB-II provides a high-depth resource for the audio-visual biometric community. By offering simultaneous multi-view data and focusing on behavioral "extreme features," it facilitates the development of more robust, real-world identification systems. The database is intended to be an open resource for the investigation of multi-view biometrics and visual speech reading.
Summary updated