HIT-AVDB-II: A New Multi-view and Extreme Feature Cases Contained Audio-Visual Database for Biometrics

Xiaoxin Lin, Hongxun Yao, Xiaopeng Hong, Qian Wang2008 · DOI

Abstract

This paper introduces HIT-AVDB-II, a multi-view audio-visual database designed to advance biometric research by capturing "extreme features"—unique habitual mouth movements—across four simultaneous camera angles and varied environmental conditions. Developed at the Harbin Institute of Technology, the database provides a robust benchmark for identifying individuals based on their unique speaking habits, moving beyond the constrained, frontal-only datasets common in the field.

Introduction

While audio-visual speech recognition and biometrics have progressed significantly, real-world application remains a challenge. Most existing databases are recorded under highly constrained conditions: subjects look directly at a single camera in perfect lighting with a limited vocabulary. In practical scenarios, head movement, lighting changes, and idiosyncratic speaking habits significantly impact system performance. HIT-AVDB-II addresses these gaps by focusing on "extreme features" (exaggerated or idiosyncratic mouth movements) which serve as a form of "visual fingerprint" for biometric identification.

The HIT-AVDB-II Database

The HIT-AVDB-II dataset consists of 30 subjects (15 male, 15 female) recorded over multiple sessions. To simulate real-world variability, recordings were conducted in three sessions (morning, afternoon, evening) against three distinct backgrounds: dark blue, light green, and dark red.

Sample frame showing the light green background condition used to simulate environmental variation.

The corpus is significantly more diverse than standard digit-based datasets, including:

  • Chinese and English poems (to capture emotion and prosody).
  • Tongue twisters (to expose fast movements and pronunciation defects).
  • The Greek alphabet and musical notes.

Recording Methodology

A sophisticated recording rig was employed to capture synchronized multi-view data. Four cameras recorded subjects simultaneously at 0° (frontal), 30°, 60°, and 90° (profile) angles. This setup allows researchers to investigate how different views complement each other, particularly the profile view, which often contains information missing from frontal perspectives.

Multi-view recording setup with four cameras at 0°, 30°, 60°, and 90°.

The Concept of "Extreme Features"

A core contribution of this work is the emphasis on "extreme features" (ext-features). The authors argue that personal habits—how individuals move their mouths uniquely or exaggerate certain sounds—are highly distinguishing characteristics. By using complex speech tasks like tongue twisters, the database intentionally triggers these idiosyncratic behaviors, providing a richer source of data for biometric systems than "proper" or neutral speech.

Experimental Protocols and Baseline

To ensure fair comparison and prevent "cherry-picking" of results, the authors established two standardized experimental protocols:

  • Protocol I (Multi-view): Evaluates algorithm performance on individual views versus various combinations of views.
  • Protocol II (Intersection): Tests session-independence by training on one day and testing on another, ensuring the system is robust to changes in lighting or subject appearance.

As a baseline, the researchers implemented a system using 2-D Discrete Cosine Transform (DCT) for feature extraction on 64x64 mouth regions, paired with a Semi-Continuous Hidden Markov Model (SCHMM) for recognition. The system utilizes the top 30 low-frequency DCT components and an SCHMM with 6 states and 8 mixtures.

Conclusion

HIT-AVDB-II provides a high-depth resource for the audio-visual biometric community. By offering simultaneous multi-view data and focusing on behavioral "extreme features," it facilitates the development of more robust, real-world identification systems. The database is intended to be an open resource for the investigation of multi-view biometrics and visual speech reading.

Summary updated

Abstract

For research on the proper law of audiovisual speech and biometrics technology, and evaluation of algorithms and systems, we construct a multi-language and multiview database HIT-AVDB-II with a corpus of various common and special sentences include Chinese and English poems, tongue twister, digits, Greek alphabet and music. The HIT-AVDB-II is ready to facilitate the investigation of multi-view biometrics technology and visual speech reading. HIT-AVDB-II contains formal and extreme feature cases for study. For fair comparison, we also establish related experiment protocols on view and intersection respectively. Further, we supplied one baseline speakeridentification recognition algorithm based on DCT to be compared.
BibTeX
@misc{lin2008hit,
  author = {Xiaoxin Lin and Hongxun Yao and Xiaopeng Hong and Qian Wang},
  title = {{HIT-AVDB-II: A New Multi-view and Extreme Feature Cases Contained Audio-Visual Database for Biometrics}},
  year = {2008},
  doi = {10.2991/jcis.2008.61},
}
Figure