BIFRNet: A Brain-Inspired Feature Restoration DNN for Partially Occluded Image Recognition

Jiahong Zhang, Lihong Cao, Qiuxia Lai, Bingyao Li, Yunxiao QinAAAI 2023 · DOI

TL;DR

This paper introduces BIFRNet, a novel deep learning architecture inspired by human visual processing, designed to tackle the challenging problem of recognizing objects that are partially occluded. By mimicking the brain's ventral and dorsal visual pathways, along with incorporating a knowledge module and a feature completion mechanism, BIFRNet achieves state-of-the-art performance on both synthetic and real-world occluded image datasets, even surpassing human-level performance in some scenarios.

What Problem Does This Paper Address?

Recognizing objects in images is a fundamental task in computer vision. However, when objects are partially hidden by occluders, standard Convolutional Neural Networks (CNNs) often struggle, leading to misclassifications. This is a significant limitation, as occlusion is ubiquitous in real-world scenarios, impacting applications like autonomous driving, surveillance, and human-computer interaction. While some methods try to extract robust features from visible parts or remove occluded features, they often fail when occlusion is severe. This paper posits that, similar to how the human brain performs "amodal completion" to infer the complete shape of an object even when parts are hidden, AI models need a similar feature restoration capability.

Methods

BIFRNet is built upon a brain-inspired architecture with three key components:

  • Visual Pathways (VVP & DVP):
    • Ventral Visual Pathway (VVP): Mimics the ventral stream in the human brain, responsible for object recognition. It uses a pre-trained VGG16 network to extract image features.
  • Dorsal Visual Pathway (DVP): Inspired by the dorsal stream, which handles spatial processing. This pathway generates a spatial attention map that highlights visible object regions and suppresses occluded ones. This is achieved through a series of dilated convolution, max-pooling, and convolution layers.
  • Knowledge Module: This module stores "prior knowledge" about object categories. It consists of an encoder that learns a knowledge representation for each image and a knowledge matrix (K) that aggregates this knowledge across categories. This module is trained to ensure that representations for different categories are well-separated.
  • Completion Module: This is the core of the feature restoration. It takes the visible features (derived from VVP and DVP) and the prior knowledge from the knowledge module to reconstruct (complete) the occluded features. This module uses a recurrent structure (ConvLSTM) to model the completion process, drawing inspiration from neuroscience findings on how the brain infers missing information.

The network is trained end-to-end, with a combined loss function that includes attention loss for DVP, KLD loss for the knowledge module, L2 loss for feature restoration, and cross-entropy loss for classification.

Datasets

The paper evaluates BIFRNet on two datasets:

  • Occluded-Vehicles Dataset: A synthetic dataset generated from PASCAL3D+ by applying four types of occluders (white boxes, noise, textures, natural objects) at four occlusion levels (0%, 20-40%, 40-60%, 60-80%).
  • Occluded-COCO Vehicles Dataset: A real-world dataset derived from MS-COCO, also featuring four occlusion levels.

Key Insights and Ideas

  • Amodal Completion for AI: The paper successfully translates the neuroscience concept of amodal completion into a deep learning framework, demonstrating its effectiveness for object recognition under occlusion.
  • Dual Pathway for Occlusion Handling: The separation of feature extraction (VVP) and occlusion discrimination (DVP) provides a more robust way to handle occluded inputs than single-pathway approaches.
  • Importance of Prior Knowledge: The knowledge module is crucial, especially for severe occlusions, highlighting that recognizing incomplete objects often relies on what we already know about them.
  • Feature-Level Restoration: Unlike previous methods that focused on image-level restoration, BIFRNet performs restoration at the feature level, which is more computationally efficient and directly aids the classification task.
  • Brain-Inspired Design Enhances Interpretability: The architecture's connection to human visual pathways makes it more interpretable, allowing researchers to understand why it works.

Key Results

  • State-of-the-Art Performance: BIFRNet significantly outperforms existing methods on both Occluded-Vehicles and Occluded-COCO Vehicles datasets across various occlusion levels.
  • Human-Like Performance: On the synthetic Occluded-Vehicles dataset, BIFRNet achieves performance close to human levels, particularly at high occlusion levels.
  • Significant Improvement on Severe Occlusion: For highly occluded images (60-80% occlusion) on the synthetic dataset, BIFRNet improves accuracy by 7% compared to the previous state-of-the-art.
  • Robustness to Unseen Occluders: The paper notes that BIFRNet is robust to occluders it has not encountered during training.

Limitations and Future Work

  • Computational Cost: While feature-level restoration is more efficient than image-level, the ConvLSTM in the completion module can still be computationally intensive.
  • Generalization to More Complex Scenarios: While BIFRNet shows robustness to unseen occluders, further research could explore its performance in more dynamic and complex real-world environments with diverse object types and occlusion patterns.
  • Further Exploration of Knowledge Module: The paper suggests that exploring different ways to form and utilize prior knowledge could lead to further improvements.
  • Broader Applications: The brain-inspired framework for occlusion discrimination, prior knowledge integration, and feature restoration could be beneficial for other POIR-like problems beyond image recognition.

Summary updated

BibTeX
@inproceedings{zhang2023bifrnet,
  author = {Jiahong Zhang and Lihong Cao and Qiuxia Lai and Bingyao Li and Yunxiao Qin},
  title = {{BIFRNet: A Brain-Inspired Feature Restoration DNN for Partially Occluded Image Recognition}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2023},
  doi = {10.1609/aaai.v37i12.26784},
}
Figure