The First Facial Landmark Tracking in-the-Wild Challenge: Benchmark and Results

Jie Shen, Stefanos Zafeiriou, Grigoris G. Chrysos, Jean Kossaifi, Georgios Tzimiropoulos, Maja PanticWorkshops @ ICCV 2015 · OpenReview

Introduction

Detecting and tracking faces in video sequences is a cornerstone of computer vision, essential for applications ranging from facial expression analysis to high-performance face recognition. While face detection—identifying a bounding box—is a mature field, facial landmark tracking involves the much more complex task of locating and following specific fiducial points (e.g., eyes, nose, mouth) as they undergo non-rigid deformations.

Historically, research has focused on static imagery, bolstered by benchmarks like the 300-W challenge. However, real-world video presents unique hurdles: temporal consistency, error accumulation over long durations, and extreme variations in pose and occlusion. This paper introduces the first comprehensive benchmark for long-term facial landmark tracking "in-the-wild," presented in conjunction with ICCV 2015.

The 300-VW Benchmark Dataset

To bridge the gap between static and temporal analysis, the authors introduced the 300-VW (300 Videos in-the-Wild) benchmark. It consists of 114 videos totaling approximately 2 hours of footage and 218,595 frames, each meticulously annotated with 68 facial landmarks. Annotating such a massive volume of data manually is prohibitive; the authors employed a semi-automatic "human-in-the-loop" process using person-specific deformable models, achieving a 4.5x speedup in annotation time without sacrificing accuracy.

Example of facial landmark tracking on historical video footage, demonstrating the dataset's diversity.

The dataset is categorized into three levels of difficulty:

  1. Category One: Controlled lighting with varied expressions but no major occlusions.
  2. Category Two: Unconstrained lighting (dark or overexposed) and arbitrary expressions.
  3. Category Three: Completely unconstrained scenarios featuring heavy occlusions (hands, objects) and extreme head poses.

Methodology and Competition Results

The challenge evaluated several participant methodologies against a baseline Supervised Descent Method (SDM). Two approaches emerged as leaders:

  • Method Yang: Utilized a spatio-temporal cascade shape regression model with a re-initialization mechanism to recover from lost tracks.
  • Method Xiao: Employed a multi-stage regression approach that prioritized "easy" landmarks (eyes, mouth) before attempting the more complex face contour.

Accuracy was quantified using the point-to-point Root-Mean-Square (RMS) error, normalized by the inter-ocular distance to ensure scale invariance.

Key Insights: Lighting vs. Occlusion

The results provided a clear picture of the current state of the art. Interestingly, top-performing algorithms showed remarkable robustness to lighting changes, performing nearly as well in Category 2 as in Category 1. This suggests that modern regression-based methods have largely solved the illumination problem.

Facial landmark tracking in the presence of partial occlusion by a microphone, a key challenge in Category 3.

However, occlusion remains the primary bottleneck. Performance dropped significantly in Category 3, where objects like microphones or hands frequently obscure parts of the face. Tracking through these "disappearing acts" and maintaining accuracy over thousands of frames remains a significant challenge for the community.

Conclusion

The 300-VW benchmark reveals that while we are reaching a performance ceiling in static image landmarking, long-term video tracking "in-the-wild" is far from solved. The transition from frontal to profile views and the handling of heavy occlusions are the next frontiers. By providing a standardized, large-scale dataset, this work sets the stage for the next generation of robust, real-time facial analysis systems.

Summary updated

Abstract

Detection and tracking of faces in image sequences is among the most well studied problems in the intersection of statistical machine learning and computer vision. Often, tracking and detection methodologies use a rigid representation to describe the facial region 1, hence they can neither capture nor exploit the non-rigid facial deformations, which are crucial for countless of applications (e.g., facial expression analysis, facial motion capture, high-performance face recognition etc.). Usually, the non-rigid deformations are captured by locating and tracking the position of a set of fiducial facial landmarks (e.g., eyes, nose, mouth etc.). Recently, we witnessed a burst of research in automatic facial landmark localisation in static imagery. This is partly attributed to the availability of large amount of annotated data, many of which have been provided by the first facial landmark localisation challenge (also known as 300-W challenge). Even though now well established benchmarks exist for facial landmark localisation in static imagery, to the best of our knowledge, there is no established benchmark for assessing the performance of facial landmark tracking methodologies, containing an adequate number of annotated face videos. In conjunction with ICCV'2015 we run the first competition/challenge on facial landmark tracking in long-term videos. In this paper, we present the first benchmark for long-term facial landmark tracking, containing currently over 110 annotated videos, and we summarise the results of the competition.
BibTeX
@inproceedings{shen2015first,
  author = {Jie Shen and Stefanos Zafeiriou and Grigoris G. Chrysos and Jean Kossaifi and Georgios Tzimiropoulos and Maja Pantic},
  title = {{The First Facial Landmark Tracking in-the-Wild Challenge: Benchmark and Results}},
  booktitle = {Workshops @ ICCV},
  year = {2015},
  url = {https://openreview.net/forum?id=1_WS3SttSiYC},
}
Figure