SuperFront: From Low-resolution to High-resolution Frontal Face Synthesis

Yu Yin, Joseph P. Robinson, Songyao Jiang, Yue Bai, Can Qin

TL;DR

SuperFront (SF-GAN) is a new generative model that synthesizes high-resolution frontal faces from low-resolution, extreme-profile images. By jointly learning super-resolution and face rotation—and supporting the fusion of multiple views via orthogonal constraints—it preserves identity far better than existing state-of-the-art methods.

What Problem Does This Paper Address?

Face frontalization—the process of computationally "rotating" a profile view to a front-facing view—is a staple of modern computer vision, essential for everything from security surveillance to digital photography. However, most existing models suffer from a "double whammy" problem:

  1. Pose Discrepancy: When a face is turned 60° or 90° away, half the features are occluded. Mapping this to a frontal view is a highly non-linear, "ill-posed" problem.
  2. Low Quality: Real-world images (like those from a distant security camera) are often low-resolution (LR). Existing frontalization models usually assume high-quality input; when fed "tiny" faces, they lose the fine details that make a person recognizable.

The authors identify that solving these two problems sequentially (upscaling then rotating, or vice versa) leads to cumulative errors. SuperFront aims to solve them collaboratively.

Methods

The core of the paper is SF-GAN (SuperFront Generative Adversarial Network). Its architecture and training regime introduce several novel elements:

  • SR-Integrated Decoder: Unlike standard GANs that just output a frontal face, SF-GAN’s decoder has two branches. One branch performs the frontalization, while a parallel Side-view Super-Resolution (SR) branch reconstructs the original profile in high resolution. This ensures the model "remembers" the fine details (like the specific shape of a nose or eye) from the input before trying to rotate them.
  • Three-Level Loss Function:
    • Pixel-level (L1): For basic color and shape consistency.
  • Patch-level (SSIM): Instead of looking at the whole face, the model compares small patches using the Structural Similarity Index (SSIM). This forces the GAN to pay attention to local structures like the periocular (eye) region and mouth.
  • Global-level: Combines adversarial loss (to make the face look "real") and an identity-preserving loss (using a pre-trained Light CNN to ensure the "fingerprint" of the face remains the same).
  • Multi-Image (MI) Fusion: The model can accept one or many LR profile shots. To make the most of multiple images, the authors use Orthogonal Regularization. This mathematical constraint penalizes redundant features, essentially forcing the model to extract different useful information from each unique camera angle.

Datasets

The authors evaluated SF-GAN on three major benchmarks:

  1. Multi-PIE: A controlled dataset with 337 subjects under various poses, illuminations, and expressions.
  2. CAS-PEAL-R1: A large-scale Chinese face database focusing on pose and pitch variations.
  3. LFW (Labeled Faces in the Wild): An unconstrained dataset used to test how the model handles real-world, messy environments.

Key Insights and Ideas

  • Collaborative vs. Sequential: The biggest "aha" moment is that super-resolution provides a "feature anchor." By forcing the model to upscale the side-view while rotating it, the side-view acts as a high-fidelity reference that prevents the frontal synthesis from becoming a generic, blurry face.
  • Diversity is Key: In the multi-image version, simply "averaging" images doesn't work well. The use of an orthogonal constraint to diversify the learned feature space is a clever way to ensure that a 30° shot and a 60° shot contribute unique value to the final output.
  • Patch-Level Matters: Global losses often result in "smooth" faces that look like mannequins. Focusing on patches ensures that high-frequency details—the tiny wrinkles and sharp edges that define identity—are preserved.

Key Results

SF-GAN consistently outperformed previous leaders in the field (like TP-GAN and M2FPA), especially on low-resolution inputs:

  • Accuracy: On the Multi-PIE dataset (Setting 2), SF-GAN achieved a Rank-1 recognition accuracy of 92.81% on LR inputs. For comparison, the previous state-of-the-art (M2FPA) achieved only 79.55% under the same conditions.
  • Visual Fidelity: Qualitatively, SF-GAN produces much sharper images. While other models produced "ghostly" or blurry features when starting from LR profiles, SF-GAN recovered distinct textures in the eyes and mouth.
  • The Power of Many: The multi-image (MI) version showed that as you add more LR inputs (from 1 to 4), the accuracy climbs significantly (e.g., from ~91% to over 99% in specific tests), proving the effectiveness of their fusion strategy.

Limitations and Future Work

While the results are impressive, a few areas remain open for exploration:

  • Extreme Occlusions: While the model handles poses up to ±90°, "in the wild" occlusions (like a hand covering the face or heavy sunglasses) are still a hurdle for most generative models, including this one.
  • Computational Load: The paper uses a deep encoder with 16 residual dense blocks. While this produces high quality, it may be computationally expensive for real-time edge applications (like mobile devices).
  • Future Directions: The authors suggest that the multi-view fusion approach could be extended to video-based face recognition, where a stream of frames provides a continuous source of "multi-image" data.

Summary updated

Abstract

Even the most impressive achievement in frontal face synthesis is challenged by large poses and low-quality data given one single side-view face. We propose a synthesizer called SuperFront GAN (SF-GAN) to accept one or more low-resolution (LR) faces at the input to then output a high-resolution (HR) frontal face with various poses and such to preserve identity information. SF-GAN includes intra-class and inter-class constraints, which allow it to learn an identity-preserving representation from multiple LR faces in an improved, comprehensive manner. We adopt an orthogonal loss as the intra-class constraint that diversifies the learned feature-space per subject. Hence, each sample is made to complement the others to its max ability. Additionally, a triplet loss is used as the inter-class constraint: it improves the discriminative power of the new representation, which, hence, maintains the identity information. Furthermore, we integrate a super-resolution (SR) side-view module as part of the SF-GAN to help preserve the finer details of HR side-views. This helps the model reconstruct the high-frequency parts of the face (i.e. periocular region, nose, and mouth regions). Quantitative and qualitative results demonstrate the superiority of SF-GAN. SF-GAN holds promise as a pre-processing step to normalize and align faces before passing to CV system for processing.
BibTeX
@inproceedings{yin2021superfront,
  author = {Yu Yin and Joseph P. Robinson and Songyao Jiang and Yue Bai and Can Qin and Yun Fu},
  title = {{SuperFront: From Low-resolution to High-resolution Frontal Face Synthesis}},
  booktitle = {ACM Multimedia},
  year = {2021},
  doi = {10.1145/3474085.3475300},
}

Loading related content...

Figure