Mitigation of Effects of Occlusion on Object Recognition with Deep Neural Networks through Low-Level Image Completion

Benjamin Chandler, Ennio MingollaComput. Intell. Neurosci 2016 · DOI

TL;DR

This paper tackles the challenge of object recognition in the presence of occlusion, a common real-world scenario that significantly degrades the performance of current deep learning models. The authors introduce a new dataset, SORBO, specifically designed to study occlusion, and propose a novel approach that uses image inpainting to "complete" occluded regions, dramatically improving recognition accuracy even with high levels of occlusion.

What Problem Does This Paper Address?

Human vision is remarkably adept at recognizing objects even when they are partially hidden. However, computer vision systems, particularly deep neural networks, struggle significantly with occlusion. This is partly because standard datasets like ImageNet and PASCAL VOC, due to biases in how images were collected and curated, contain relatively few occluded objects. This lack of data makes it difficult to train models that are robust to occlusion, and also hinders researchers' ability to accurately measure and understand how occlusion impacts recognition performance. The paper aims to bridge this gap by providing a dataset that explicitly addresses occlusion and developing a method to mitigate its negative effects.

Methods

The core of the paper's methodological contribution lies in two areas:

  • The SORBO Dataset: The authors introduce the Synthetic Object Recognition Benchmark with Occlusions (SORBO) dataset. SORBO is built upon the existing NORB dataset, which is known for its rich parametric metadata (object class, camera angle, lighting, etc.). SORBO extends NORB by adding various types (bars, blobs, random noise) and levels of occlusion, along with ground-truth segmentation masks for each occluded image. This detailed metadata allows for precise measurement of occlusion's impact and the effectiveness of mitigation strategies.
  • Inpainting for Occlusion Mitigation: The proposed solution involves a preprocessing step that uses image inpainting. Inpainting is a technique that reconstructs missing or damaged parts of an image. In this context, the algorithm uses the ground-truth segmentation masks to identify occluded pixels and then "inpaints" these regions, effectively filling them in with plausible content that is consistent with the surrounding visible parts of the image. This inpainted image is then fed into the classification network. The paper explores both ground-truth segmentation and an automated segmentation approach.

Datasets

  • NORB (Normalized-Uniform Dataset): Used as the base for SORBO. It's a parametric dataset with rich metadata.
  • SORBO (Synthetic Object Recognition Benchmark with Occlusions): The newly introduced dataset. It extends NORB with various types and levels of occlusion and provides ground-truth segmentation masks. SORBO aims to enable precise measurement of occlusion robustness and the development of occlusion-resistant algorithms.

Key Insights and Ideas

  • Dataset Bias is a Major Hurdle: The paper strongly emphasizes how biases in existing large-scale datasets (like PASCAL VOC and ImageNet) mask the significant problem of occlusion for object recognition. This bias leads to an overestimation of current models' real-world robustness.
  • Occlusion is a Fundamental Challenge: Even state-of-the-art convolutional neural networks (ConvNets) show a dramatic drop in performance as occlusion levels increase, performing only marginally better than random chance in highly occluded scenarios.
  • Inpainting is a Powerful Mitigation Strategy: Image inpainting, when combined with accurate segmentation, can effectively restore recognition performance even with high levels of occlusion. This suggests that if we can "see through" occlusions, recognition becomes much more feasible.
  • Inpainting as Data Augmentation: Surprisingly, using inpainting with ground-truth masks not only improved performance on occluded images but also led to better performance on unoccluded test images. This suggests that the inpainting process, by creating "disoccluded" training data, acts as a form of data augmentation that reduces overfitting and improves generalization.
  • Segmentation Quality is Crucial: The performance of the inpainting-based mitigation heavily relies on the accuracy of the segmentation masks. While ground-truth masks yield excellent results, automated segmentation, even if imperfect, can still achieve parity with the base system on heavily occluded images, but it sacrifices the generalization benefits seen with ground-truth data.

Key Results

  • A standard ConvNet trained on unoccluded data degrades to near chance-level accuracy (around 20%) with 81-100% occlusion.
  • Training with a mix of occluded and unoccluded data (the "combined" training set) improves robustness but still results in significant performance degradation.
  • The inpainting preprocessing stage, using ground-truth segmentation, allowed a ConvNet to achieve performance nearly as good on heavily occluded images (81-100% occlusion) as on unoccluded images.
  • The inpainting approach with ground-truth masks improved performance on unoccluded test images by approximately 5-10% compared to a model trained only on unoccluded data.
  • Automated segmentation, while not as effective as ground-truth, still allowed the inpainting method to perform comparably to the "occluded training" condition on high-occlusion images.

Limitations and Future Work

  • Segmentation Accuracy: The paper acknowledges that the performance of the inpainting method is critically dependent on the quality of the segmentation masks. The automated segmentation algorithm used in the paper has limitations, particularly at the borders of occlusions, leading to false alarms and misses. Future work would involve developing more robust segmentation algorithms.
  • Generalization of Inpainting: While the paper demonstrates the effectiveness of inpainting on the SORBO dataset, its performance on more complex, real-world images with less controlled occlusions would need further investigation.
  • Computational Cost: While the paper uses a fast linear diffusion inpainting algorithm for efficiency, more sophisticated inpainting methods might yield even better results but at a higher computational cost.
  • Dataset Complexity: The SORBO dataset's occlusions are deliberately designed to be easy to segment. Applying similar mitigation strategies to datasets with more complex, textured, or object-based occlusions would be a significant challenge.

Summary updated

BibTeX
@article{chandler2016mitigation,
  author = {Benjamin Chandler and Ennio Mingolla},
  title = {{Mitigation of Effects of Occlusion on Object Recognition with Deep Neural Networks through Low-Level Image Completion}},
  journal = {Comput. Intell. Neurosci.},
  year = {2016},
  doi = {10.1155/2016/6425257},
}
Figure