Boosting Deep Face Recognition via Disentangling Appearance and Geometry

Ali Dabouei, Fariborz Taherkhani, Sobhan Soleymani, Jeremy Dawson, Nasser NasrabadiWACV 2020

TL;DR

This paper introduces a novel framework for disentangling facial appearance and geometry in deep learning models for face recognition. By generating geometrically identical faces through spatial transformations, the approach enhances feature learning and augments the training set, leading to significant improvements in face recognition accuracy.

What Problem Does This Paper Address?

Face recognition (FR) is a critical task in computer vision, but it faces challenges due to variations in pose, illumination, expression, and background. While deep learning has revolutionized FR, achieving high accuracy in unconstrained environments remains difficult. Existing methods often focus on improving loss functions or network architectures. This paper tackles the problem by proposing to disentangle two fundamental aspects of a face: its appearance (e.g., skin texture, hair color) and its geometry (e.g., facial structure, pose). The hypothesis is that by learning these aspects separately, the model can develop more robust and discriminative representations, leading to better recognition performance.

Methods

The core of the proposed method, termed Disentangling the Appearance and Geometry (DAG), involves creating pairs of images that isolate either appearance or geometry variations.

  1. Geometrically Identical Faces: For an input face image ($x_i$), the method finds a "closest neighbor" face ($x_{iv}$) from a different identity in the geometry space. This is achieved by comparing landmark locations. Then, a spatial transformation (Thin Plate Spline - TPS) is applied to $x_i$ to create a new image ($x_i'$) that has the geometry of $x_i$ but the appearance of $x_{iv}$. This process effectively generates geometrically similar but appearance-different pairs. Conversely, by mapping $x_i$ to a face with similar appearance but different geometry, appearance-similar pairs are created.
  2. Disentangling Networks: Two networks are trained to learn separate representations for geometry ($g(x)$) and appearance ($a(x)$). A third network ($f$) then combines these representations to produce the final embedding for face recognition.
  3. Loss Functions:
    • Geometry-Preserving Loss ($L_g$): This loss encourages the geometry representations of geometrically similar faces to be close, while ensuring they are dissimilar to faces with different landmark disparities. It uses cosine similarity and an angular margin.
  4. Appearance-Preserving Loss ($L_a$): This loss encourages the appearance representations of geometrically transformed faces (which share appearance) to be close.
  5. Identification Loss ($L_{id}$): A standard angular-margin based loss (like SphereFace or CosFace) is used to ensure the final combined representation is discriminative for identification.

The total loss function combines these three components with hyper-parameter scaling ($\lambda_a$, $\lambda_g$).

Datasets

  • Training: CASIA-WebFace, VGGFace2, and a private dataset were used for training. A "quasi-augmented" dataset was also created by combining MS-Celeb-1M and CASIA-WebFace.
  • Evaluation: Labeled Faces in the Wild (LFW), YouTube Faces (YTF), and MegaFace Challenge 1 (MF1) were used for face recognition benchmarks. CelebA was used for facial attribute prediction.

Key Insights and Ideas

  • Geometry is Crucial for Identification: The paper's experiments reveal that geometry representations alone provide more informative features for face identification than appearance representations, likely because facial structure is more intrinsically tied to identity than transient appearance characteristics.
  • Appearance Captures Semantic Attributes: While geometry is key for identity, appearance representations capture rich semantic attributes like hair color, skin tone, and even the presence of accessories like glasses or hats.
  • Disentanglement Augments Training Data: The generation of geometrically identical faces effectively augments the training set, providing more diverse samples for the model to learn from.
  • Improved Feature Learning in Early Layers: The disentanglement process forces earlier convolutional layers to learn more representative features that are invariant to appearance (for geometry) or geometry (for appearance), leading to better overall representation quality.
  • Knowledge Transferability: The disentangled representations learned for face recognition can be effectively transferred to other tasks like facial attribute prediction, demonstrating their general utility.

Key Results

  • Face Recognition Performance Boost: Integrating DAG with state-of-the-art methods like SphereFace and CosFace consistently improved performance on LFW, YTF, and MegaFace. For example, CosFace+DAG achieved 99.81% accuracy on LFW and 98.0% on YTF.
  • Attribute Prediction Improvement: Model A (weakly supervised) trained with DAG improved attribute prediction by an average of 8.34% on CelebA compared to baselines. Model B (fine-tuned) outperformed baselines on 16 out of 18 attributes.
  • Hyper-parameter Sensitivity: The performance was sensitive to the weighting of the appearance ($\lambda_a$) and geometry ($\lambda_g$) losses, with optimal values found through experimentation. The model was more sensitive to $\lambda_g$, suggesting that matching geometry representations is more challenging.

Limitations and Future Work

  • Landmark Disparity: The paper acknowledges that the geometric transformation might introduce slight mismatches due to the limited number of landmarks used. This could explain why matching geometry representations is harder.
  • Future Work: The authors suggest that task-specific considerations for the training phase can further improve the performance of deep learning models, hinting at exploring more sophisticated ways to leverage disentangled representations. They also implicitly suggest further research into optimizing the hyper-parameters for the disentanglement losses.

Summary updated

BibTeX
@inproceedings{dabouei2020boosting,
  author = {Ali Dabouei and Fariborz Taherkhani and Sobhan Soleymani and Jeremy Dawson and Nasser Nasrabadi},
  title = {{Boosting Deep Face Recognition via Disentangling Appearance and Geometry}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2020},
}
Figure