Cross-Pose Face Recognition — A Virtual View Generation Approach Using Clustering Based LVTM

Xi Li, Tomokazu Takahashi, Daisuke Deguchi, Ichiro Ide, Hiroshi MuraseIEICE 2013 · DOI

Abstract

This paper presents an approach for cross-pose face recognition by virtual view generation using an appearance clustering based local view transition model (c-LVTM). Previously, the traditional global pattern based view transition model (VTM) method was extended to its local version called LVTM, which learns the linear transformation of pixel values between frontal and non-frontal image pairs from training images using partial image in a small region for each location, instead of transforming the entire image pattern. In this paper, we show that the accuracy of the appearance transition model and the recognition rate can be further improved by better exploiting the inherent linear relationship between frontal-nonfrontal face image patch pairs. This is achieved based on the observation that variations in appearance caused by pose are closely related to the corresponding 3D structure and intuitively frontal-nonfrontal patch pairs from more similar local 3D face structures should have a stronger linear relationship. Thus for each specific location, instead of learning a common transformation as in the LVTM, the corresponding local patches are first clustered based on an appearance similarity distance metric and then the transition models are learned separately for each cluster. In the testing stage, each local patch for the input non-frontal probe image is transformed using the learned local view transition model corresponding to the most visually similar cluster. The experimental results on a real-world face dataset demonstrated the superiority of the proposed method in terms of recognition rate.

The Cross-Pose Challenge

Face recognition systems often struggle with "cross-pose" scenarios—where the database (gallery) contains a single frontal "passport" photo, but the person being identified (the probe) is looking away from the camera. The core challenge is that the visual difference between the same person viewed from two different angles is often greater than the difference between two different people viewed from the same angle. Traditional methods like "Eigenfaces" fail here because they cannot bridge that gap. While 3D morphable models exist to solve this, they are computationally expensive and require high-resolution images to accurately map facial landmarks, making them impractical for low-resolution surveillance footage.

Methods: From VTM to c-LVTM

The authors build upon the View Transition Model (VTM), evolving it through three stages:

  1. VTM (Global): Learns a single linear transformation to turn an entire side-view image into a frontal one.
  2. LVTM (Local): Instead of the whole face, it breaks the image into local patches and learns a specific transformation for each location (e.g., a "nose patch" transform).
  3. c-LVTM (Clustering-based Local VTM): The authors observed that even in the same location (like the corner of the eye), different people have different 3D structures. A single "eye corner" transform is insufficient.
The c-LVTM pipeline showing clustering of local patches and the transition model application.

The Innovation: Clustering Local Patches

The c-LVTM approach operates in two primary phases:

  • Training: Training images are divided into patches. For every patch location, K-means clustering groups patches that look similar. A unique linear transformation is then learned for each cluster at each location.
  • Testing: When a new side-view image is processed, it is broken into patches. For each patch, the system finds the most visually similar cluster from the training set and applies that specific transformation.
Visualization of local patches extracted from a face image for localized transformation.

Finally, all transformed patches are stitched back together (averaging the overlaps) to create a virtual frontal face, which is then fed into a standard recognition engine.

Key Insights and Results

  • Appearance as a Proxy for 3D Structure: The "aha moment" is the realization that 2D appearance similarity is a strong indicator of 3D geometric similarity. By grouping similar-looking patches, the model implicitly accounts for the 3D shape of the face without needing a complex 3D mesh.
  • Linearity in Locality: While the relationship between a side-view and a front-view face is non-linear globally, it becomes highly linear when zooming in on small, structurally similar patches.
  • Visual Fidelity and Accuracy: In a 5-fold cross-validation on the SOFTPIA JAPAN dataset (250 individuals), c-LVTM consistently outperformed its predecessors. At a 50-degree angle—a very difficult case—c-LVTM maintained a significantly higher recognition rate while other methods' performance plummeted.

Limitations and Future Work

The clustering and searching process adds a higher computational burden compared to simpler models, taking about 3 minutes to process a test image versus 0.5 seconds for standard LVTM. Additionally, the model currently has low robustness to significant lighting changes. Future work will focus on optimizing computation, handling subtle facial expressions, and improving tolerance for errors in automatic landmark detection.

Summary updated

Abstract

This paper presents an approach for cross-pose face recognition by virtual view generation using an appearance clustering based local view transition model. Previously, the traditional global pattern based view transition model (VTM) method was extended to its local version called LVTM, which learns the linear transformation of pixel values between frontal and non-frontal image pairs from training images using partial image in a small region for each location, instead of transforming the entire image pattern. In this paper, we show that the accuracy of the appearance transition model and the recognition rate can be further improved by better exploiting the inherent linear relationship between frontal-nonfrontal face image patch pairs. This is achieved based on the observation that variations in appearance caused by pose are closely related to the corresponding 3D structure and intuitively frontal-nonfrontal patch pairs from more similar local 3D face structures should have a stronger linear relationship. Thus for each specific location, instead of learning a common transformation as in the LVTM, the corresponding local patches are first clustered based on an appearance similarity distance metric and then the transition models are learned separately for each cluster. In the testing stage, each local patch for the input non-frontal probe image is transformed using the learned local view transition model corresponding to the most visually similar cluster. The experimental results on a real-world face dataset demonstrated the superiority of the proposed method in terms of recognition rate.
BibTeX
@inproceedings{li2013cross,
  author = {Xi Li and Tomokazu Takahashi and Daisuke Deguchi and Ichiro Ide and Hiroshi Murase},
  title = {{Cross-Pose Face Recognition — A Virtual View Generation Approach Using Clustering Based LVTM}},
  booktitle = {IEICE},
  year = {2013},
  doi = {10.1587/transinf.e96.d.531},
}
Figure