Compositional Convolutional Networks For Robust Object Classification under Occlusion

Adam Kortylewski, Qing Liu, Huiyu Wang, Zhishuai Zhang, Alan L. Yuille

TL;DR

This paper introduces a hybrid approach that combines the high discriminative power of Deep Convolutional Neural Networks (DCNNs) with the inherent robustness of compositional models. By representing objects as a spatial arrangement of "parts" derived from DCNN features, the model can successfully classify objects under heavy, unseen occlusion—a scenario where standard deep networks typically fail.

What Problem Does This Paper Address?

While DCNNs have revolutionized object classification, they possess a significant "Achilles' heel": they do not generalize well to partial occlusions or "mask attacks" (where parts of an image are intentionally hidden). Because it is impossible to train a network on every possible occlusion pattern, DCNNs often misclassify occluded objects with high confidence or fail entirely when faced with dataset bias.

In contrast, compositional models represent objects as a collection of parts and their spatial relationships. This makes them naturally robust to occlusion (if a part is missing, the rest of the model still fits), but they generally lack the raw classification accuracy of deep learning on clean, non-occluded images. The authors seek to bridge this gap, creating a system that is both highly accurate and robust.

Object classification under occlusion: (a) DCNN fails, (b) Compositional model succeeds by identifying parts and ignoring occluded regions.

Methods

The authors propose a two-branch architecture that integrates DCNNs and compositional models:

  1. Feature Extraction & Dictionary Learning: They train a standard DCNN (like VGG-16) and extract feature maps from intermediate layers (e.g., pool4). These high-dimensional features are clustered using k-means to create a "dictionary" of parts.
  2. Generative Part Model: They model the spatial distribution of these parts for each object class using a Bernoulli distribution. This captures where certain features (like a wheel or a window) are likely to appear.
  3. Mixture Models for 3D: To handle changes in 3D pose or viewpoint, they use a "mixture of compositional models," where different components of the mixture represent different viewpoints of the same object.
  4. Occlusion Modeling: They explicitly add an occlusion model that treats occluders as independent background clutter. This allows the model to "deactivate" parts of the object model that are inconsistent with the image data.
  5. Inference Strategy: At runtime, the DCNN processes the image first. If the DCNN is uncertain (low prediction probability), the system switches to the compositional branch to perform a robust verification.

Datasets

The model was evaluated on two primary benchmarks, both modified to include synthetic and natural occlusions:

  • MNIST: The classic handwritten digit dataset, augmented with "mask attacks" (white boxes, noise, or textures covering 20-80% of the digit).
  • PASCAL3D+: A dataset of vehicles in natural scenes. The authors simulated occlusion using noise/textures and by superimposing segmented objects from the VehicleSemanticPart dataset to mimic real-world occluders.

Key Insights and Ideas

  • DCNN Features as Semantic Parts: A key "aha moment" is that intermediate DCNN features already act as excellent part detectors. By clustering them, the authors gain the representational power of deep learning without the "black box" fragility.
  • Explainability by Design: Unlike standard DCNNs, this model can explain why it made a decision. It can point to the specific parts it detected and highlight the regions it believes are occluded.
  • Generalization without Training on Occlusion: The most significant finding is that the model recognizes occluded objects even if it was only trained on non-occluded images. It generalizes to the concept of occlusion rather than memorizing specific occlusion patterns.
Explainability: The model localizes detected parts (left/middle) and generates an occlusion map (right) to justify its prediction.

Key Results

  • PASCAL3D+: The combined model outperformed a standard VGG-16 by 13.9% on partially occluded objects while maintaining nearly identical performance (98.3% vs 98.6%) on clean images.
  • MNIST: The model showed an absolute performance gain of 19.4% over the DCNN for occluded digits.
  • Robustness to Bias: When the training data was biased (e.g., occlusions only appearing on the right side of images), the DCNN failed to recognize occlusions on the left during testing. The proposed hybrid model, however, generalized across the entire image regardless of training bias.

Limitations and Future Work

  • Threshold Dependency: The system relies on a manually set uncertainty threshold ($\tau$) to decide when to switch from the DCNN to the compositional model.
  • Computational Overhead: Running two branches, especially the generative compositional model, adds computational complexity compared to a simple feedforward DCNN.
  • Integration: While the branches are combined at inference, the authors suggest that future work could involve more tightly integrating these compositional principles directly into the DCNN architecture (Compositional Convolutional Networks) to allow for end-to-end training.

Summary updated

Abstract

Deep convolutional neural networks (DCNNs) are powerful models that yield impressive results at object classification. However, recent work has shown that they do not generalize well to partially occluded objects and to mask attacks. In contrast to DCNNs, compositional models are robust to partial occlusion, however, they are not as discriminative as deep models. In this work, we combine DCNNs and compositional object models to retain the best of both approaches: a discriminative model that is robust to partial occlusion and mask attacks. Our model is learned in two steps. First, a standard DCNN is trained for image classification. Subsequently, we cluster the DCNN features into dictionaries. We show that the dictionary components resemble object part detectors and learn the spatial distribution of parts for each object class. We propose mixtures of compositional models to account for large changes in the spatial activation patterns (e.g. due to changes in the 3D pose of an object). At runtime, an image is first classified by the DCNN in a feedforward manner. The prediction uncertainty is used to detect partially occluded objects, which in turn are classified by the compositional model. Our experimental results demonstrate that combining compositional models and DCNNs resolves a fundamental problem of current deep learning approaches to computer vision: The combined model recognizes occluded objects, even when it has not been exposed to occluded objects during training, while at the same time maintaining high discriminative performance for non-occluded objects.
BibTeX
@inproceedings{kortylewski2020compositional,
  author = {Adam Kortylewski and Qing Liu and Huiyu Wang and Zhishuai Zhang and Alan L. Yuille},
  title = {{Compositional Convolutional Networks For Robust Object Classification under Occlusion}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2020},
  doi = {10.1109/wacv45572.2020.9093560},
}
Figure