Robust LSTM-Autoencoders for Face De-Occlusion in the Wild

Fang Zhao, Jiashi Feng, Jian Zhao, Wenhan Yang, Shuicheng Yan

TL;DR

The authors present a Robust LSTM-Autoencoder (RLA) designed to "de-occlude" faces—removing obstacles like sunglasses, masks, or hands—to improve recognition in unconstrained environments. By combining a multi-scale spatial encoder with a dual-channel recurrent decoder and adversarial training, the model progressively restores facial details while preserving the person's unique identity.

What Problem Does This Paper Address?

Face recognition has come a long way, but it still struggles "in the wild" when faces are partially covered. Occlusions like a hand over a mouth or large sunglasses aren't just random noise; they are "spatially contiguous," meaning they block out entire functional regions of the face.

Most existing solutions only work in "closed-set" scenarios (where the model already knows the people it's looking at) or handle only one specific type of occlusion. This paper tackles face de-occlusion in the wild, where the system must handle unknown individuals and diverse, unpredictable occlusions in a single, robust framework.

Methods

The RLA framework breaks the problem into a sequence of sub-tasks rather than trying to "fix" the image in one pass.

  1. Multi-scale Spatial LSTM Encoder: Instead of a standard CNN, the model uses a Spatial LSTM. It "reads" the face in patches (e.g., top-left to bottom-right). By using multiple scales (local patches + a coarse version of the whole image), the encoder learns to ignore occluded patches and focus on visible features.
  2. Dual-Channel LSTM Decoder: The decoder has two parallel paths that collaborate:
  3. Reconstruction Channel: Generates the occlusion-free face.
  4. Detection Channel: Identifies exactly which pixels are occluded.
  5. Recurrent Refinement: The decoder runs for 8 steps. In each step, it uses the detection map to decide which parts of the original image to keep and which generated parts to "fill in," progressively sharpening the result.
  6. Identity-Preserving RLA (IP-RLA): To prevent the "blurry average face" problem common in standard autoencoders, the authors add two extra components:
  7. Supervised CNN: Ensures the recovered face still contains the discriminative features needed to identify the person.
  8. Adversarial CNN (GAN): A discriminator that pushes the model to generate realistic textures rather than blurry blobs.

Datasets

  • CASIA-WebFace (Training): The authors used ~380,000 near-frontal faces and synthetically added 9 types of occlusions (sunglasses, masks, hands, glasses, eye masks, scarfs, phones, books, and cups) to create a massive training set.
  • LFW (Testing): The standard Labeled Faces in the Wild dataset was used for evaluation, with synthetic occlusions added to test generalization to new identities.
  • 50OccPeople (Testing): A newly introduced dataset containing real-world occlusions (not synthetic) across 50 subjects and 1,200 images to verify performance in practical scenarios.

Key Insights and Ideas

  • Recurrence is Key: The paper argues that de-occlusion is too complex for a "one-stop" solution. By using an LSTM to iteratively refine the face, the model can "reason" about facial symmetry and structure more effectively.
  • Detection Aids Reconstruction: By explicitly modeling where the occlusion is (the detection channel), the model avoids "corrupting" the parts of the face that were already visible and clear.
  • Beyond Pixel-Matching: The authors realized that minimizing Mean Squared Error (MSE) alone leads to blurry results. Integrating an identity loss and an adversarial loss is crucial for making the output useful for actual face recognition systems.

Key Results

  • Recognition Accuracy: On the occluded LFW dataset, the IP-RLA model significantly reduced the Equal Error Rate (EER). For example, with "hand" occlusions, the EER dropped to 5.6%, outperforming standard Autoencoders (26.8%) and PCA-based methods (12.4%).
  • Qualitative Superiority: Visual results show that while other methods (like SSDA or SRC) produce blurry or distorted faces, RLA recovers sharp features like eyes and mouths that look remarkably similar to the ground truth.
  • Real-World Performance: On the 50OccPeople dataset (real occlusions), the model achieved an 18.0% EER, proving that a model trained on synthetic data can generalize to real-world "wild" conditions.

Limitations and Future Work

  • Hallucination vs. Reality: When a face is heavily occluded (e.g., the entire lower half), the model must "hallucinate" the missing parts. While the result looks like a realistic face, it may not perfectly match the actual hidden features of that specific individual.
  • Frontal Bias: The current work focuses on near-frontal faces (-45° to +45°). Handling extreme profile views with occlusions remains a challenge.
  • Computational Cost: Running an LSTM for 8 iterations is more computationally expensive than a single forward pass of a standard CNN, which might impact real-time surveillance applications.

Summary updated

Abstract

Face recognition techniques have been developed significantly in recent years. However, recognizing faces with partial occlusion is still challenging for existing face recognizers, which is heavily desired in real-world applications concerning surveillance and security. Although much research effort has been devoted to developing face de-occlusion methods, most of them can only work well under constrained conditions, such as all of faces are from a pre-defined closed set of subjects. In this paper, we propose a robust LSTM-Autoencoders (RLA) model to effectively restore partially occluded faces even in the wild. The RLA model consists of two LSTM components, which aims at occlusion-robust face encoding and recurrent occlusion removal respectively. The first one, named multi-scale spatial LSTM encoder, reads facial patches of various scales sequentially to output a latent representation, and occlusion-robustness is achieved owing to the fact that the influence of occlusion is only upon some of the patches. Receiving the representation learned by the encoder, the LSTM decoder with a dual channel architecture reconstructs the overall face and detects occlusion simultaneously, and by feat of LSTM, the decoder breaks down the task of face de-occlusion into restoring the occluded part step by step. Moreover, to minimize identify information loss and guarantee face recognition accuracy over recovered faces, we introduce an identity-preserving adversarial training scheme to further improve RLA. Extensive experiments on both synthetic and real data sets of faces with occlusion clearly demonstrate the effectiveness of our proposed RLA in removing different types of facial occlusion at various locations. The proposed method also provides significantly larger performance gain than other de-occlusion methods in promoting recognition performance over partially-occluded faces.
BibTeX
@article{zhao2018robust,
  author = {Fang Zhao and Jiashi Feng and Jian Zhao and Wenhan Yang and Shuicheng Yan},
  title = {{Robust LSTM-Autoencoders for Face De-Occlusion in the Wild}},
  journal = {IEEE Trans. Image Process.},
  year = {2018},
  doi = {10.1109/tip.2017.2771408},
}
Figure