TL;DR
This paper introduces a hybrid approach that combines the high discriminative power of Deep Convolutional Neural Networks (DCNNs) with the inherent robustness of compositional models. By representing objects as a spatial arrangement of "parts" derived from DCNN features, the model can successfully classify objects under heavy, unseen occlusion—a scenario where standard deep networks typically fail.
What Problem Does This Paper Address?
While DCNNs have revolutionized object classification, they possess a significant "Achilles' heel": they do not generalize well to partial occlusions or "mask attacks" (where parts of an image are intentionally hidden). Because it is impossible to train a network on every possible occlusion pattern, DCNNs often misclassify occluded objects with high confidence or fail entirely when faced with dataset bias.
In contrast, compositional models represent objects as a collection of parts and their spatial relationships. This makes them naturally robust to occlusion (if a part is missing, the rest of the model still fits), but they generally lack the raw classification accuracy of deep learning on clean, non-occluded images. The authors seek to bridge this gap, creating a system that is both highly accurate and robust.
Methods
The authors propose a two-branch architecture that integrates DCNNs and compositional models:
- Feature Extraction & Dictionary Learning: They train a standard DCNN (like VGG-16) and extract feature maps from intermediate layers (e.g.,
pool4). These high-dimensional features are clustered using k-means to create a "dictionary" of parts. - Generative Part Model: They model the spatial distribution of these parts for each object class using a Bernoulli distribution. This captures where certain features (like a wheel or a window) are likely to appear.
- Mixture Models for 3D: To handle changes in 3D pose or viewpoint, they use a "mixture of compositional models," where different components of the mixture represent different viewpoints of the same object.
- Occlusion Modeling: They explicitly add an occlusion model that treats occluders as independent background clutter. This allows the model to "deactivate" parts of the object model that are inconsistent with the image data.
- Inference Strategy: At runtime, the DCNN processes the image first. If the DCNN is uncertain (low prediction probability), the system switches to the compositional branch to perform a robust verification.
Datasets
The model was evaluated on two primary benchmarks, both modified to include synthetic and natural occlusions:
- MNIST: The classic handwritten digit dataset, augmented with "mask attacks" (white boxes, noise, or textures covering 20-80% of the digit).
- PASCAL3D+: A dataset of vehicles in natural scenes. The authors simulated occlusion using noise/textures and by superimposing segmented objects from the VehicleSemanticPart dataset to mimic real-world occluders.
Key Insights and Ideas
- DCNN Features as Semantic Parts: A key "aha moment" is that intermediate DCNN features already act as excellent part detectors. By clustering them, the authors gain the representational power of deep learning without the "black box" fragility.
- Explainability by Design: Unlike standard DCNNs, this model can explain why it made a decision. It can point to the specific parts it detected and highlight the regions it believes are occluded.
- Generalization without Training on Occlusion: The most significant finding is that the model recognizes occluded objects even if it was only trained on non-occluded images. It generalizes to the concept of occlusion rather than memorizing specific occlusion patterns.
Key Results
- PASCAL3D+: The combined model outperformed a standard VGG-16 by 13.9% on partially occluded objects while maintaining nearly identical performance (98.3% vs 98.6%) on clean images.
- MNIST: The model showed an absolute performance gain of 19.4% over the DCNN for occluded digits.
- Robustness to Bias: When the training data was biased (e.g., occlusions only appearing on the right side of images), the DCNN failed to recognize occlusions on the left during testing. The proposed hybrid model, however, generalized across the entire image regardless of training bias.
Limitations and Future Work
- Threshold Dependency: The system relies on a manually set uncertainty threshold ($\tau$) to decide when to switch from the DCNN to the compositional model.
- Computational Overhead: Running two branches, especially the generative compositional model, adds computational complexity compared to a simple feedforward DCNN.
- Integration: While the branches are combined at inference, the authors suggest that future work could involve more tightly integrating these compositional principles directly into the DCNN architecture (Compositional Convolutional Networks) to allow for end-to-end training.
Summary updated