TL;DR
This paper presents a real-time system for detecting head nods and shakes using infrared pupil tracking and Hidden Markov Models (HMMs). The system achieves a respectable 78.46% accuracy, offering a foundation for more natural human-computer interaction.
What Problem Does This Paper Address?
Non-verbal communication, particularly head gestures like nods and shakes, plays a crucial role in human interaction, conveying agreement, disagreement, emotions, and conversational cues. However, accurately detecting these subtle, rapid, and sometimes jerky movements in real-time using computer vision is challenging. Existing methods often struggle with the required precision or speed, especially under varying lighting conditions or when users wear accessories like glasses. This paper aims to bridge this gap by developing a robust and efficient system for real-time head nod and shake detection.
Methods
The core of the system relies on two main components:
- Infrared Pupil Tracking: An infrared camera equipped with synchronized infrared LEDs is used to capture images. The LEDs create an "red-eye effect" (on-axis LEDs) and provide general illumination (off-axis LEDs). By subtracting two interlaced images (one with on-axis LEDs on, one with off-axis LEDs on), a difference image is generated where pupils appear as distinct bright or dark spots. This difference image is then processed to detect and track pupil positions.
- Hidden Markov Model (HMM) Based Pattern Analyzer: The system infers head movement direction (up, down, left, right, or none) based on the change in pupil positions between consecutive frames. These movement directions are then converted into observation symbols. Two HMMs, one for head nods and one for head shakes, are trained using these observation sequences. Each HMM has three states and five observation symbols. During testing, the system compares the likelihood of the observed sequence under each HMM to classify the gesture.
A key aspect of their approach is the feature extraction module, which operates in either a "tracking mode" (when pupil information is available) or a "detection mode" (when pupil information is lost, requiring a broader search). They also employ spatial constraints to validate detected pupil regions, making the tracking more robust to noise and artifacts.
Datasets
The authors collected a custom dataset for training and testing. This dataset consists of natural head nods and shakes from ten subjects (five male, five female, two wearing glasses) who were asked to answer factual questions posed by a "Microsoft agent." This approach was chosen to elicit more natural gestures compared to simply asking subjects to nod or shake their head on command. The dataset comprises 110 samples (62 head nods, 48 head shakes), collected under varied lighting conditions and with subjects exhibiting various facial expressions. A random 40% of head nods and 40% of head shakes were used for training, with the remaining data used for testing.
Key Insights and Ideas
- Infrared Pupil Tracking for Robustness: Using infrared light and LEDs significantly improves pupil detection and tracking, making the system less susceptible to variations in ambient lighting compared to visible light cameras.
- HMMs for Temporal Pattern Recognition: HMMs are well-suited for modeling sequential data like head movements, allowing the system to learn the temporal patterns characteristic of nods and shakes.
- "Between-Eyes" Region vs. Pupil Tracking: While some prior work focused on the "between-eyes" region, this paper highlights the advantage of directly tracking pupils for more precise movement inference.
- Handling Real-World Complexity: The system is designed to handle natural, unexaggerated gestures, varied lighting, and even some facial expressions, which are common in real-world human-computer interaction scenarios.
- Challenges with Glasses: The specular nature of glasses can interfere with pupil tracking, leading to missed detections or false positives. However, the authors note an interesting case where a specular point on glasses was successfully tracked.
Key Results
- Real-time Performance: The system operates in real-time at 30 frames per second (fps).
- Recognition Accuracy: The system achieved a combined recognition accuracy of 78.46% on the test dataset.
- Recognition Rate for Head Nods: 81.08%
- Recognition Rate for Head Shakes: 75.0%
- No Confusion Between Nods and Shakes: The authors report no confusion between head nods and head shakes, indicating that the system effectively distinguishes between vertical and horizontal head movements.
- Training vs. Testing Performance: The system performed better on the training set (93.34% combined recognition rate) than on the test set, which is typical and suggests good generalization capabilities.
Limitations and Future Work
- Limitations:
- Glasses Interference: The presence of glasses can significantly degrade performance due to specular reflections.
- Subject Closing Eyes: If a subject closes their eyes during a gesture, detection can be missed.
- False Positives: Some head nods were misclassified when subjects laughed with rhythmic up-and-down head movements.
- Future Work: The authors suggest extending the system to include:
- Expression recognition.
- Facial feature extraction.
- Pose estimation.
- Integration of these components to create a system that recognizes affect more broadly.
Summary updated