Robust Real-Time Face Tracking and Gesture Recognition

J. Heizmann, Alexander ZelinskyIJCAI 1997

Abstract

This paper presents a robust, real-time system for face tracking and gesture recognition developed at the Australian National University. Operating at 30Hz without markers or specialized lighting, the system utilizes a network of interdependent Kalman filters to maintain tracking of facial features. A novel "virtual third eye" mechanism allows the system to recover from tracking failures and occlusions within frames. The architecture supports the recognition of 12 distinct gestures through a two-layer approach combining atomic action recognition with Finite State Machines (FSMs). Experimental results demonstrate high recognition rates for common gestures and resilience to head rotations up to 60 degrees.

Introduction

In the mid-1990s, human-robot interaction was significantly hindered by the fragility and latency of computer vision systems. Traditional template matching often failed when subjects moved or when features were occluded by hands or other objects. This research addresses these limitations by developing a "self-healing" tracking system capable of real-time performance in unconstrained environments, providing a more natural interface for human-machine communication.

Methodology

The system architecture combines specialized hardware with a novel "buddy system" for facial features. It employs hardware-accelerated correlation for high-speed template matching using a mean absolute error method.

Central to its robustness is the use of interdependent Kalman filters. Rather than tracking features like eyes, nose, and mouth in isolation, the system models the face as a 2D grid. Each feature is tracked by its own Kalman filter, but these filters share information. If the system loses track of one feature, it uses the positions of the others and the known geometry of the human face to estimate the missing feature's location.

Facial feature tracking using an interdependent 2D grid.

To handle recovery from total occlusion, the system spawns "Area search windows" when confidence drops. A "virtual third eye" search window is used to verify geometric consistency, preventing the system from misidentifying features (e.g., swapping the left and right eye) during re-acquisition. Gesture recognition is implemented in two layers: a lower layer identifies "atomic actions" (basic movements), while an upper layer uses Finite State Machines (FSMs) to recognize complex sequences as specific gestures.

Examples of recognized head gestures and their corresponding motion vectors.

Experimental Results

The system was evaluated using in-house video sequences and live real-time testing. It successfully maintained a processing rate of 30Hz, matching the NTSC frame rate. Key performance metrics include:

  • Robustness: The system handled head rotations of up to 60 degrees and recovered from total occlusion within a few frames.
  • Gesture Accuracy: Simple gestures such as "yes" (nodding) and "no" (shaking) achieved a 95% recognition rate. Complex "turn" gestures were recognized with over 70% accuracy.
  • Detection Sensitivity: Short gestures like blinks and winks were consistently detected, though they exhibited a 20% false-positive rate due to their brief duration.

Conclusion and Future Work

By prioritizing probabilistic fusion and geometric constraints over simple binary matching, the researchers created a significantly more robust interface for human-machine interaction. The use of FSMs allowed for complex gesture recognition without the high computational cost of traditional methods like dynamic time warping.

Future work aims to extend the current 2D model to a 3D adaptive model to handle full profile views and improve performance under varying illumination conditions. Integration into assistive robotics for the elderly and disabled remains a primary goal for this technology.

Summary updated

BibTeX
@inproceedings{heizmann1997robust,
  author = {J. Heizmann and Alexander Zelinsky},
  title = {{Robust Real-Time Face Tracking and Gesture Recognition}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {1997},
}
Figure