TL;DR
This paper argues that while deep convolutional networks have revolutionized feature extraction for object detection, the region-wise classifiers have been largely overlooked, often relying on simple MLPs. The authors introduce "Networks on Convolutional feature maps" (NoCs) and demonstrate that deep, convolutional classifiers are crucial for achieving state-of-the-art object detection performance, even when using powerful pre-trained feature extractors like ResNets.
What Problem Does This Paper Address?
Modern object detection pipelines typically consist of two main parts: a feature extractor and an object classifier. While feature extractors have seen tremendous advancements with deep convolutional neural networks (CNNs), the object classifier component has lagged behind. Many successful detectors, such as SPPnet and Fast/Faster R-CNN, employ relatively simple multi-layer perceptrons (MLPs) as classifiers on top of rich convolutional features. This paper questions whether this simplified approach to classification is optimal and investigates the impact of designing more sophisticated, deep, and convolutional classifiers for object detection tasks. The authors aim to bridge the gap between the advancements in feature extraction and the often-underdeveloped classification stage.
Methods
The core contribution of this paper is the introduction of Networks on Convolutional feature maps (NoCs). These are region-wise classifier networks that operate on shared, region-independent convolutional features extracted from the entire image. The process involves:
- Feature Extraction: Using pre-trained deep CNNs (like VGG or ZF) to generate convolutional feature maps for the entire image.
- Region Proposal: Generating region proposals (e.g., using Selective Search).
- RoI Pooling: Extracting fixed-size feature maps for each proposed region from the shared convolutional feature maps.
- NoC Classification: Feeding these fixed-size region features into a novel NoC architecture for classification.
The NoC architectures explored include:
- MLP-based NoCs: Varying the depth of fully-connected layers.
- ConvNet-based NoCs: Incorporating convolutional layers within the classifier itself, allowing for region-aware feature learning.
- Maxout NoCs: Using the maxout activation function to improve scale selection from multi-scale feature maps.
The paper also investigates the impact of fine-tuning these NoC architectures versus random initialization, and the benefits of using deeper convolutional feature extractors versus deeper convolutional classifiers.
Datasets
The paper uses the following datasets:
- PASCAL VOC 2007: Used extensively for experiments, with both the original 5k training images and an augmented 16k set (VOC 2007 + VOC 2012 trainval).
- PASCAL VOC 2012: Used as part of the augmented training set for PASCAL VOC 2007 experiments.
- MS COCO: Used for experiments with more challenging scenarios, including models like ResNets and GoogleNets, with 80 categories.
Key Insights and Ideas
- Deep Classifiers Matter: The paper strongly suggests that the design of the region-wise classifier is as critical as the feature extractor for achieving high object detection accuracy. Simply using powerful feature extractors doesn't automatically translate to top performance if the classifier is weak.
- Convolutional Classifiers Outperform MLPs: Introducing convolutional layers within the classifier (NoC) significantly improves performance over standard MLPs, indicating that region-aware convolutional features are more effective for detection.
- Orthogonal Benefits: Deeper convolutional classifiers provide benefits that are complementary to deeper feature maps. This means that improving both the feature extractor and the classifier leads to greater gains than improving only one.
- Fine-tuning vs. Random Initialization: While fine-tuning pre-trained models is beneficial, randomly initialized deep NoCs can still achieve surprisingly good results, highlighting the power of deep network architectures for learning from detection data.
- NoCs are Essential for Modern Architectures: For "fully convolutional" models like ResNets and GoogleNets, which lack traditional fully-connected layers, designing effective NoCs is not just an improvement but a necessity to achieve competitive detection performance.
Key Results
- PASCAL VOC 2007:
- A randomly initialized 4-layer MLP NoC achieved 53.6% mAP, a 7.8% improvement over a simple SVM on RoI features.
- A convolutional NoC (2conv3fc) trained on VOC 07+12 trainval achieved 58.9% mAP, outperforming a 3fc NoC baseline (56.5% mAP).
- Fine-tuned NoCs significantly boosted performance, with a maxout 2conv3fc NoC on VGG-16 achieving 68.8% mAP (without bounding box regression).
- When combined with bounding box regression, their best NoC achieved 71.6% mAP, competitive with then-state-of-the-art methods.
- MS COCO:
- Using a deeper MLP NoC (3fc) with ResNet-101 improved AP from 21.3% to 26.3% (with stride 16).
- A convolutional NoC with ResNet-101 further improved AP to 27.2%.
- The paper demonstrates that even with powerful ResNet features, a naïve Faster R-CNN with a shallow classifier performs poorly, underscoring the importance of the NoC.
Limitations and Future Work
- Limitations:
- The paper primarily focuses on the classifier aspect, assuming a fixed feature extractor. While they experiment with different feature extractors (ZF, VGG, ResNet), the core innovation is in the classifier.
- Some experiments are conducted under restricted conditions (e.g., frozen feature extractors, post-hoc SVMs) to isolate the effects of NoCs, though they note these findings generally hold when relaxed.
- Future Work:
- The authors suggest that their findings are essential for improving Faster R-CNN with modern, fully convolutional architectures like ResNets.
- The insights gained from NoCs can be applied to other detection frameworks and tasks.
- The paper implicitly points towards further exploration of novel convolutional classifier architectures.
Summary updated