Regressive Tree Structured Model for Facial Landmark Localization

Gee-Sern Hsu, Kai-Hsiang Chang, Shih-Chieh HuangICCV 2015 · DOI

Introduction

Facial landmark localization—the precise identification of points such as eye corners, nose tips, and mouth edges—is a fundamental task in computer vision. While the Tree Structured Model (TSM) has been a popular choice for its ability to unify face detection, pose estimation, and landmarking, its practical utility has been historically limited by high computational costs and sensitivity to image scale. The Regressive Tree Structured Model (RTSM) addresses these bottlenecks, offering a high-speed alternative that maintains and even improves the robustness of the original framework.

The Challenge: Speed and Scale

Standard TSM implementations are notoriously slow, often requiring 25 to 50 seconds to process a single image. This latency makes them unsuitable for real-time processing or scenarios involving multiple faces. Furthermore, TSMs struggle with small faces (e.g., 40x40 pixels) because high-density landmark patches tend to overlap at low resolutions, leading to feature ambiguity and detection failure.

The RTSM Framework

The RTSM introduces a multi-stage pipeline designed to optimize both speed and accuracy through a coarse-to-fine strategy and regression-based landmarking:

  1. Coarse-to-Fine Search: The process begins with a Coarse TSM (c-TSM) on low-resolution image octaves (1/4 scale) to rapidly identify face candidates using a sparse set of landmarks. This is followed by a Refined TSM (r-TSM) at mid-resolution (1/2 scale) to establish pose and a better initial fit.
  2. Bilateral Support Vector Regressor (BSVR): To avoid the slow part-based search for every individual landmark, RTSM employs a dual-regression strategy. The Forward BSVR maps sparse landmarks from the r-TSM to a full, dense set. The Backward BSVR then identifies and corrects landmarks with high localization errors by "relocating" them based on global shape traits.
Figure 3: The model identifies key facial parts and their spatial relationships across different poses, including profile and frontal views.

Key Insights

A primary innovation of RTSM is the realization that "less is more" for small or low-resolution faces. By using fewer landmarks with larger relative patches in the coarse stage, the model avoids the feature crowding that plagues standard TSMs. Additionally, the shift from part-based search to shape-based regression (BSVR) provides a significant "shortcut," allowing the model to infer the positions of dense landmarks much faster than traditional methods.

Figure 5: Dense landmark localization on a profile view, demonstrating the model's ability to infer hidden or occluded features using regression.

Results and Performance

Evaluation on benchmarks like Multi-PIE, LFPW (Labeled Face Parts in the Wild), and AFW (Annotated Faces in the Wild) demonstrates the effectiveness of the RTSM:

  • 100x Speedup: On the Multi-PIE dataset, processing time dropped from 25 seconds to just 0.25 seconds per face.
  • Enhanced Accuracy: Localization accuracy on the LFPW dataset improved from 61.4% to 71.5% within a 5% error margin.
  • Robustness to Occlusion: The bilateral refinement process allows the model to self-correct, pulling misaligned landmarks back into plausible positions even in challenging profile views where half the face may be hidden.

Conclusion

The Regressive Tree Structured Model successfully transforms a powerful but slow framework into a practical tool for modern computer vision. By combining multi-resolution search with efficient bilateral regression, RTSM achieves significant gains in both speed and accuracy. While 0.25 seconds per face is a massive improvement, future work aims to further reduce latency for real-time high-frame-rate video and integrate more advanced feature extractors beyond HOG (Histogram of Oriented Gradients).

Summary updated

BibTeX
@inproceedings{hsu2015regressive,
  author = {Gee-Sern Hsu and Kai-Hsiang Chang and Shih-Chieh Huang},
  title = {{Regressive Tree Structured Model for Facial Landmark Localization}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2015},
  doi = {10.1109/iccv.2015.439},
}
Figure