Discriminative Face Hallucination via Locality-Constrained and Category Embedding Representation

Licheng Liu, Rushi Lan, Yaonan Wang

Abstract

Recent years have witnessed the rapid development of face image hallucination techniques. However, previous face hallucination methods are largely unsupervised and ignore the label information of training samples, leading to undesirable results. This article proposes a locality-constrained and category embedding representation (LCER) method to super-resolve face images in a supervised manner by embedding label information into the data representation. The proposed LCER incorporates locality priors and category information into a unified framework, aiming to leverage the advantages of locality in preserving the true topological structure of the data manifold alongside discriminability in exposing class subspace information. This strategy allows LCER not only to preserve sharper image details but also to guarantee that face structure patterns are transferred mainly from the same subject during super-resolution reconstruction. Extensive experiments demonstrate that LCER achieves superior face hallucination performance in both quantitative measurements and visual impressions compared to several state-of-the-art methods.

Introduction

Face hallucination is a specialized super-resolution (SR) task that reconstructs high-resolution (HR) facial images from low-resolution (LR) inputs. Because human faces are highly structured, generic SR methods often fail to capture the fine-grained details necessary for recognition. A major gap in existing state-of-the-art methods is their unsupervised nature; by treating all training samples equally and ignoring identity labels, these methods often produce "ghosting" artifacts where facial features from different individuals are blended together. This results in a face that may look clear but no longer resembles the original subject.

Methodology: The LCER Framework

The proposed LCER framework is a position-patch-based approach. Since facial features like eyes, noses, and mouths generally appear in consistent relative locations, the model operates on small, overlapping patches at specific coordinates. The core of LCER is a unified objective function with three main components:

  1. Data Fidelity: Ensures the reconstructed patch closely matches the input LR patch.
  2. Locality Regularization: Based on manifold learning, this term encourages the model to represent the input using its "nearest neighbors" in the training set, preserving local geometric structures and fine textures.
  3. Category Embedding: This supervised term penalizes reconstructions that deviate from the subspace of the subject's specific category (identity). It ensures that the "hallucinated" details are primarily transferred from the same individual.
Figure 2: Conceptual illustration of Category Embedding, where the reconstruction is constrained by the specific identity class of the subject.

Crucially, the authors provide a closed-form solution for this optimization problem using Lagrange multipliers, making it computationally efficient. The complexity is $O(nK^2 + (C+2)K^3)$, where $K$ is the number of training samples and $C$ is the number of classes. For cases where the input identity is unknown, an initial label estimation step is employed using Locality-constrained Linear Coding (LLC).

Experimental Results

The model was evaluated on the AR Dataset (700 images) and the NUST RWFR Dataset (2,400 images), the latter featuring challenging unaligned faces with varied expressions and lighting.

Quantitative Performance

In tests on the AR dataset at 4x downsampling, LCER achieved a PSNR of 29.98 dB and an SSIM of 89.09%, significantly outperforming the baseline LSR (28.65 dB / 86.72%). The model also maintained robustness when subjected to Gaussian noise or extreme downsampling (up to 16x).

Visual Quality

Qualitative results indicate that LCER produces sharper facial features, such as clearer eyes and teeth, with fewer artifacts compared to unsupervised manifold learning methods. By focusing on patches at specific coordinates, the model leverages structural consistency to learn class-specific features more effectively than holistic methods.

Figure 4: A sample of a hallucinated face patch using the LCER method, demonstrating the preservation of sharp, identity-consistent details.

Conclusion

LCER demonstrates that incorporating identity labels into the reconstruction process is vital for high-fidelity face hallucination. By combining the "how" of locality (texture) with the "who" of category embedding (identity), the framework ensures results are both sharp and identity-consistent. Future research may explore integrating these category embedding principles into deep learning architectures like GANs to combine generative power with identity-preserving constraints.

Summary updated

Abstract

Recent years have witnessed the rapid development of face image hallucination techniques. However, the previous face hallucination methods are unsupervised and ignore the label information of training samples, leading to undesirable results. This article proposes a locality-constrained and category embedding representation (LCER) method to super-resolve face image in a supervised manner by embedding the label information in data representation. The proposed LCER incorporates the locality prior and category information into one unified framework, which aims to learn both the advantages of locality in preserving the true typologic structure of data manifold and the discriminability in exposing the class subspace information. Such strategy allows the LCER not only to preserve more sharpen image details but also to guarantee the face structure pattern be transferred mainly from the same subject in super-resolution reconstruction. Extensive experiments were conducted to evaluate the proposed LCER, and the comparative results demonstrate that it achieved superior face hallucination performance in both the quantitative measurements and visual impressions compared to several state-of-the-art.
BibTeX
@article{liu2020discriminative,
  author = {Licheng Liu and Rushi Lan and Yaonan Wang},
  title = {{Discriminative Face Hallucination via Locality-Constrained and Category Embedding Representation}},
  journal = {IEEE Transactions on Systems Man and Cybernetics Systems},
  year = {2020},
  doi = {10.1109/tsmc.2020.2965572},
}
Figure