TL;DR
This paper proposes a novel image representation method that combines the strengths of traditional local descriptors and deep neural network activations. By extracting multi-scale dense activations from pre-trained CNNs and aggregating them using a modified Fisher kernel with scale-wise normalization, they achieve significant performance improvements in visual recognition tasks.
What Problem Does This Paper Address?
Traditional image recognition methods often rely on hand-crafted local descriptors (like SIFT) which are good at capturing geometric invariance but lack rich semantic information. More recently, deep Convolutional Neural Networks (CNNs) have shown remarkable success by learning powerful, hierarchical representations directly from data. However, these deep activations, while semantically rich, are often sensitive to geometric variations (scale, rotation, etc.), limiting their robustness. This paper aims to bridge this gap by creating a more robust and discriminative image representation that leverages both the semantic richness of CNN activations and the geometric invariance properties of traditional methods.
Methods
The core of the proposed method, termed Multi-scale Pyramid Pooling (MPP), involves several key steps:
- Efficient Multi-scale Dense Activation Extraction: Instead of processing scaled versions of an image through a standard CNN, the authors modify the fully connected layers of a pre-trained CNN into equivalent convolutional layers. This allows for the efficient extraction of dense local activations across multiple scales from a single input image without redundant computations.
- Fisher Kernel Aggregation: The extracted multi-scale dense activations are treated as local descriptors. These are then aggregated using the Fisher kernel framework. The Fisher kernel is known for its ability to capture higher-order statistics beyond simple pooling, leading to more discriminative representations.
- Scale-wise Normalization: A crucial modification to the standard Fisher kernel is the introduction of a scale-wise normalization step. This addresses the observation that CNN activations have different scale properties compared to traditional descriptors. By normalizing activations across different scales before aggregation, the method becomes more robust to geometric variations.
- Dimensionality Reduction and Pooling: Before Fisher encoding, Principal Component Analysis (PCA) is used to reduce the dimensionality of the activation vectors. After obtaining Fisher vectors for each scale, they are merged into a single global vector using average pooling, followed by power and L2 normalization.
Datasets
The proposed method was evaluated on three standard visual recognition benchmarks:
- MIT Indoor 67: Used for scene classification.
- PASCAL VOC 2007: Used for object classification.
- Oxford 102 Flowers: Used for fine-grained object classification.
The authors used pre-trained CNN models (AlexNet and CNNS) trained on ImageNet for extracting features.
Key Insights and Ideas
- CNN activations are semantically rich but geometrically sensitive: While deep features capture high-level semantics, they don't inherently possess the geometric invariance of traditional local features.
- Multi-scale processing is key for CNNs too: Just as traditional descriptors benefit from multi-scale analysis, so do CNN activations for achieving better geometric robustness.
- Scale-wise normalization is critical for Fisher kernels with CNNs: Directly applying Fisher kernels to raw CNN activations without considering their scale-dependent properties leads to suboptimal performance. A specific normalization strategy is essential.
- Efficient dense activation extraction is feasible: Modifying fully connected layers to convolutional ones allows for rapid extraction of multi-scale local features, making the overall approach computationally viable.
- Fisher kernels offer advantages over simpler pooling: The paper demonstrates that Fisher kernels, when properly adapted, provide a more discriminative representation than average pooling or concatenation of activations.
Key Results
The proposed Multi-scale Pyramid Pooling (MPP) method demonstrated significant performance improvements across the evaluated datasets:
- MIT Indoor 67: +17.76% accuracy compared to a single activation vector.
- PASCAL VOC 2007: +7.18% mAP compared to a single activation vector.
- The method also showed superior performance compared to other multi-scale pooling strategies like VLAD pooling and concatenation.
- The approach also enabled the generation of object confidence maps for weakly-supervised object detection/localization, even without bounding box annotations during training.
Limitations and Future Work
- Computational Cost: While the dense activation extraction is made efficient, the overall process still involves multiple scales and Fisher kernel computation, which can be computationally intensive.
- Fine-tuning: The authors primarily use pre-trained CNNs without fine-tuning on the target datasets. They suggest that further improvements could be achieved by fine-tuning the CNNs, using target data augmentation, or incorporating ground truth bounding boxes.
- Future Directions: The authors suggest that their representation can be further improved by combining it with other complementary features or by exploring different pooling strategies. The ability to generate object confidence maps opens avenues for object detection and localization tasks.
Summary updated