Multitask Learning Strategy with Pseudo-Labeling: Face Recognition, Facial Landmark Detection, and Head Pose Estimation

Yongju Lee, Sungjun Jang, Han Byeol Bae, Taejae Jeon, Sangyoun Lee2024 · DOI

TL;DR

This paper tackles the challenge of training robust facial analysis models for real-world scenarios, where data is often limited and noisy. They introduce a novel multitask learning framework that leverages pseudo-labeling to create a more diverse and comprehensive training dataset, leading to state-of-the-art performance in face recognition, facial landmark detection, and head pose estimation.

What Problem Does This Paper Address?

Many facial analysis tasks, like face recognition, facial landmark detection, and head pose estimation, perform well in controlled lab settings but struggle in real-world applications. This performance gap is largely due to the difficulty in acquiring large, diverse, and accurately labeled datasets. Specifically, tasks like facial landmark detection and head pose estimation often suffer from a lack of diverse face identities (IDs) and background noise. While face recognition datasets are more abundant, they may not cover the full range of poses and variations needed for other tasks. Existing methods often rely on large, well-curated datasets, which are expensive and time-consuming to create, especially for tasks requiring precise annotations. This paper aims to bridge this gap by developing a method that can learn effectively from less ideal, real-world data.

Methods

The core of this work is a multitask learning framework that combines pseudo-labeling with an integrated network architecture.

  • Pseudo-Labeling: This technique is used to generate labels for unlabeled or partially labeled data. The authors leverage existing pre-trained models for face recognition (FR), facial landmark detection (FLD), and head pose estimation (HPE). They then use these models to predict labels for a large dataset (VGGFace2, MS1MV3, and 300W-LP). Crucially, they employ a sophisticated sampling and fitting algorithm (Algorithm 1) to refine these pseudo-labels, especially for landmarks, by considering multiple augmented views of an image and incorporating an eye blink detection network to handle variations like closed eyes. For face IDs, they implement a duplicate ID verification process to create a clean, unduplicated set of labels.
  • Multitask Learning: The pseudo-labeled dataset is then used to train a single network that performs all three tasks simultaneously. The network architecture is based on a modified IResNet50, where the first three layers are shared across all tasks, and the final layer is split into task-specific sublayers. This shared architecture promotes knowledge transfer between tasks.
  • Novel Loss Functions: To ensure balanced learning and prevent one task from dominating, they introduce a weighted total loss function. For FLD, they combine the Wing loss with a mean square error (MSE) loss, regularized by a term ($\lambda_P$) that helps in the initial learning phase. For FR, they propose a "large pose margin cosine loss" that adapts the margin based on the pose ratio, aiming for pose-invariant recognition.

Datasets

The authors utilize a combination of existing datasets for their framework:

  • Training Data:
    • VGGFace2 [28]: Used for face recognition, providing a large number of images and diverse backgrounds.
  • MS1MV3 [29]: Also used for face recognition, contributing to noise and alignment.
  • 300W-LP [22]: A key dataset for pseudo-labeling, containing labels for HPE and FLD.
  • Evaluation Datasets:
    • Face Recognition: LFW [65], CFP-FP [70], AgeDB [71], and IJB-C [31].
  • Head Pose Estimation: AFLW2000-3D [22] and BIWI [42].
  • Facial Landmark Detection: AFLW2000-3D [22].

Key Insights and Ideas

  • Pseudo-labeling as a data augmentation strategy: The paper demonstrates that pseudo-labeling, when done carefully with robust algorithms, can effectively create a large, diverse training dataset that compensates for the lack of ground truth labels and limited IDs in existing datasets.
  • Multitask learning for complementary benefits: By training FR, FLD, and HPE together, the model learns features that are beneficial across tasks. For instance, pose information from HPE can inform FLD, and robust features learned for FR can aid in pose-invariant recognition.
  • Pose-invariant face recognition: The proposed "large pose margin cosine loss" is a significant contribution, directly addressing the challenge of recognizing faces across a wide range of poses, which is a common failure point for standard FR models.
  • Handling eye variations in FLD: The integration of an eye blink detection network to refine landmark predictions is a clever way to improve robustness in FLD, particularly for challenging cases like closed eyes.
  • Unified framework for diverse facial analysis: The success across three distinct but related tasks suggests the potential for this framework to be extended to other facial analysis problems, highlighting its versatility.

Key Results

  • State-of-the-Art Performance: The proposed multitask learning model achieved SOTA or near-SOTA performance on several benchmarks:
  • Head Pose Estimation: Achieved SOTA on AFLW2000-3D (MAE 3.32) and BIWI (MAE 3.54).
  • Facial Landmark Detection: Achieved near-SOTA on AFLW2000-3D, with particularly strong performance in lower pose variations.
  • Face Recognition: Demonstrated competitive performance on IJB-C, achieving 94.17% at TAR@FAR = 1 × 10−4, and strong results on LFW, CFP-FP, and AgeDB.
  • Robustness: The model showed stable performance even with the presence of outliers in the pseudo-labeled dataset.
  • Efficiency: Despite performing multitask computations, the proposed method achieved competitive frames per second (FPS) compared to other works, even when using a single GPU.

Limitations and Future Work

  • Outliers in Pseudo-Labeling: While the framework is robust to outliers, the paper acknowledges that the pseudo-labeling process, especially for large pose variations, may still introduce some inaccuracies. This is particularly noted in the FLD results for higher pose ranges.
  • Dataset Limitations: The authors suggest that further refinement of the pseudo-labeled training dataset throughout the entire framework could lead to even better performance.
  • Future Work: The paper implies that the framework's success in these three tasks suggests its applicability to a broader range of facial analysis problems, indicating potential for future extensions.

Summary updated

BibTeX
@misc{lee2024multitask,
  author = {Yongju Lee and Sungjun Jang and Han Byeol Bae and Taejae Jeon and Sangyoun Lee},
  title = {{Multitask Learning Strategy with Pseudo-Labeling: Face Recognition, Facial Landmark Detection, and Head Pose Estimation}},
  year = {2024},
  doi = {10.3390/s24103212},
}
Figure