Abstract
Recent years have witnessed the rapid development of face image hallucination techniques. However, previous face hallucination methods are largely unsupervised and ignore the label information of training samples, leading to undesirable results. This article proposes a locality-constrained and category embedding representation (LCER) method to super-resolve face images in a supervised manner by embedding label information into the data representation. The proposed LCER incorporates locality priors and category information into a unified framework, aiming to leverage the advantages of locality in preserving the true topological structure of the data manifold alongside discriminability in exposing class subspace information. This strategy allows LCER not only to preserve sharper image details but also to guarantee that face structure patterns are transferred mainly from the same subject during super-resolution reconstruction. Extensive experiments demonstrate that LCER achieves superior face hallucination performance in both quantitative measurements and visual impressions compared to several state-of-the-art methods.
Introduction
Face hallucination is a specialized super-resolution (SR) task that reconstructs high-resolution (HR) facial images from low-resolution (LR) inputs. Because human faces are highly structured, generic SR methods often fail to capture the fine-grained details necessary for recognition. A major gap in existing state-of-the-art methods is their unsupervised nature; by treating all training samples equally and ignoring identity labels, these methods often produce "ghosting" artifacts where facial features from different individuals are blended together. This results in a face that may look clear but no longer resembles the original subject.
Methodology: The LCER Framework
The proposed LCER framework is a position-patch-based approach. Since facial features like eyes, noses, and mouths generally appear in consistent relative locations, the model operates on small, overlapping patches at specific coordinates. The core of LCER is a unified objective function with three main components:
- Data Fidelity: Ensures the reconstructed patch closely matches the input LR patch.
- Locality Regularization: Based on manifold learning, this term encourages the model to represent the input using its "nearest neighbors" in the training set, preserving local geometric structures and fine textures.
- Category Embedding: This supervised term penalizes reconstructions that deviate from the subspace of the subject's specific category (identity). It ensures that the "hallucinated" details are primarily transferred from the same individual.
Crucially, the authors provide a closed-form solution for this optimization problem using Lagrange multipliers, making it computationally efficient. The complexity is $O(nK^2 + (C+2)K^3)$, where $K$ is the number of training samples and $C$ is the number of classes. For cases where the input identity is unknown, an initial label estimation step is employed using Locality-constrained Linear Coding (LLC).
Experimental Results
The model was evaluated on the AR Dataset (700 images) and the NUST RWFR Dataset (2,400 images), the latter featuring challenging unaligned faces with varied expressions and lighting.
Quantitative Performance
In tests on the AR dataset at 4x downsampling, LCER achieved a PSNR of 29.98 dB and an SSIM of 89.09%, significantly outperforming the baseline LSR (28.65 dB / 86.72%). The model also maintained robustness when subjected to Gaussian noise or extreme downsampling (up to 16x).
Visual Quality
Qualitative results indicate that LCER produces sharper facial features, such as clearer eyes and teeth, with fewer artifacts compared to unsupervised manifold learning methods. By focusing on patches at specific coordinates, the model leverages structural consistency to learn class-specific features more effectively than holistic methods.
Conclusion
LCER demonstrates that incorporating identity labels into the reconstruction process is vital for high-fidelity face hallucination. By combining the "how" of locality (texture) with the "who" of category embedding (identity), the framework ensures results are both sharp and identity-consistent. Future research may explore integrating these category embedding principles into deep learning architectures like GANs to combine generative power with identity-preserving constraints.
Summary updated