Abstract
Recognition of human communication has previously focused on deliberately acted emotions or in structured or artificial social contexts. This makes the result hard to apply to realistic social situations. This paper describes the recording of spontaneous human communication in a specific and common social situation: conversation between two people. The clips are then annotated by multiple observers to reduce individual variations in interpretation of social signals. Temporal and static features are generated from tracking using heuristic and algorithmic methods. Optimal features for classifying examples of spontaneous communication signals are then extracted by AdaBoost. The performance of the boosted classifier is comparable to human performance for some communication signals, even on this challenging and realistic data set.
Introduction
Facial feature tracking is a cornerstone of computer vision, essential for everything from emotion recognition to driver fatigue monitoring. However, it faces three major hurdles: appearance variation due to pose, tracking drift over time, and the massive data requirements of traditional high-accuracy methods like Active Appearance Models. This research presents a method for tracking facial landmarks and head pose simultaneously without requiring a pre-trained model of a specific person's appearance, providing a robust foundation for the analysis of non-verbal communication in natural settings.
Methodology
The proposed system follows a "detect, track, and learn" loop to maintain stability across wide head rotations without the drift typically associated with self-learning algorithms.
Initialization and 3D Constraints
The system starts by identifying 19 easy-to-find landmarks (eyes, nose, mouth) on the first frame. It fits a generic 3D face mesh to these points and then "back-projects" a total of 51 feature points onto the image. This 3D model acts as a structural constraint, preventing the trackers from wandering off into the background.
Online Template Learning
As the user moves their head, the system captures small image patches (templates) for each feature. If the head reaches a new angle, a new template is stored. This creates a library of "looks" for every feature at different orientations, allowing the system to re-acquire features when the head returns to a previous pose.
Weighted RANSAC Pose Estimation
To estimate the head's 3D pose, we use a Weighted RANSAC approach with the Levenberg-Marquardt (LM) algorithm. By integrating confidence weights directly into the RANSAC sampling process, the system becomes significantly more resilient to outliers, such as shadows or partial occlusions that occur during natural conversation.
Experimental Results
The tracking system was evaluated using the Boston University (BU) head pose dataset, consisting of 45 video sequences featuring 5 different subjects with electromagnetic sensor ground truth.
- Pose Accuracy: The average error was 3.9° for pitch, 3.1° for roll, and 4.2° for yaw.
- Tracking Precision: The average 2D feature tracking error was 3.7 pixels.
- Robustness: The system successfully tracked subjects through significant rotations where traditional single-template trackers would typically fail.
Following the generation of these temporal and static features, an AdaBoost-based feature selection process was employed to identify optimal signals for classifying spontaneous communication. The performance of the resulting boosted classifier is comparable to human performance for several communication signals, even on challenging, realistic data.
Discussion and Future Work
The integration of a generic 3D mesh as a structural anchor is the key insight preventing tracking drift. While the appearance is learned online, the geometry remains fixed. However, the reliance on a rigid mesh can be a limitation for subjects whose face shapes differ significantly from the model. Future work will focus on incorporating variable shape models to better fit individual users and handle non-rigid expressive movements like talking or smiling, which are prevalent in natural human interaction.
Summary updated