TL;DR
The authors present a Robust LSTM-Autoencoder (RLA) designed to "de-occlude" faces—removing obstacles like sunglasses, masks, or hands—to improve recognition in unconstrained environments. By combining a multi-scale spatial encoder with a dual-channel recurrent decoder and adversarial training, the model progressively restores facial details while preserving the person's unique identity.
What Problem Does This Paper Address?
Face recognition has come a long way, but it still struggles "in the wild" when faces are partially covered. Occlusions like a hand over a mouth or large sunglasses aren't just random noise; they are "spatially contiguous," meaning they block out entire functional regions of the face.
Most existing solutions only work in "closed-set" scenarios (where the model already knows the people it's looking at) or handle only one specific type of occlusion. This paper tackles face de-occlusion in the wild, where the system must handle unknown individuals and diverse, unpredictable occlusions in a single, robust framework.
Methods
The RLA framework breaks the problem into a sequence of sub-tasks rather than trying to "fix" the image in one pass.
- Multi-scale Spatial LSTM Encoder: Instead of a standard CNN, the model uses a Spatial LSTM. It "reads" the face in patches (e.g., top-left to bottom-right). By using multiple scales (local patches + a coarse version of the whole image), the encoder learns to ignore occluded patches and focus on visible features.
- Dual-Channel LSTM Decoder: The decoder has two parallel paths that collaborate:
- Reconstruction Channel: Generates the occlusion-free face.
- Detection Channel: Identifies exactly which pixels are occluded.
- Recurrent Refinement: The decoder runs for 8 steps. In each step, it uses the detection map to decide which parts of the original image to keep and which generated parts to "fill in," progressively sharpening the result.
- Identity-Preserving RLA (IP-RLA): To prevent the "blurry average face" problem common in standard autoencoders, the authors add two extra components:
- Supervised CNN: Ensures the recovered face still contains the discriminative features needed to identify the person.
- Adversarial CNN (GAN): A discriminator that pushes the model to generate realistic textures rather than blurry blobs.
Datasets
- CASIA-WebFace (Training): The authors used ~380,000 near-frontal faces and synthetically added 9 types of occlusions (sunglasses, masks, hands, glasses, eye masks, scarfs, phones, books, and cups) to create a massive training set.
- LFW (Testing): The standard Labeled Faces in the Wild dataset was used for evaluation, with synthetic occlusions added to test generalization to new identities.
- 50OccPeople (Testing): A newly introduced dataset containing real-world occlusions (not synthetic) across 50 subjects and 1,200 images to verify performance in practical scenarios.
Key Insights and Ideas
- Recurrence is Key: The paper argues that de-occlusion is too complex for a "one-stop" solution. By using an LSTM to iteratively refine the face, the model can "reason" about facial symmetry and structure more effectively.
- Detection Aids Reconstruction: By explicitly modeling where the occlusion is (the detection channel), the model avoids "corrupting" the parts of the face that were already visible and clear.
- Beyond Pixel-Matching: The authors realized that minimizing Mean Squared Error (MSE) alone leads to blurry results. Integrating an identity loss and an adversarial loss is crucial for making the output useful for actual face recognition systems.
Key Results
- Recognition Accuracy: On the occluded LFW dataset, the IP-RLA model significantly reduced the Equal Error Rate (EER). For example, with "hand" occlusions, the EER dropped to 5.6%, outperforming standard Autoencoders (26.8%) and PCA-based methods (12.4%).
- Qualitative Superiority: Visual results show that while other methods (like SSDA or SRC) produce blurry or distorted faces, RLA recovers sharp features like eyes and mouths that look remarkably similar to the ground truth.
- Real-World Performance: On the 50OccPeople dataset (real occlusions), the model achieved an 18.0% EER, proving that a model trained on synthetic data can generalize to real-world "wild" conditions.
Limitations and Future Work
- Hallucination vs. Reality: When a face is heavily occluded (e.g., the entire lower half), the model must "hallucinate" the missing parts. While the result looks like a realistic face, it may not perfectly match the actual hidden features of that specific individual.
- Frontal Bias: The current work focuses on near-frontal faces (-45° to +45°). Handling extreme profile views with occlusions remains a challenge.
- Computational Cost: Running an LSTM for 8 iterations is more computationally expensive than a single forward pass of a standard CNN, which might impact real-time surveillance applications.
Summary updated