Abstract
In recent years, deep architectures have been used for transfer learning with state-of-the-art performance in many datasets. The properties of their features remain, however, largely unstudied under the transfer perspective. In this work, we present an extensive analysis of the resiliency of feature vectors extracted from deep models, with special focus on the trade-off between performance and compression rate. By introducing perturbations to image descriptions extracted from a deep convolutional neural network, we change their precision and number of dimensions, measuring how it affects the final score. We show that deep features are more robust to these disturbances when compared to classical approaches, achieving a compression rate of 98.4%, while losing only 0.88% of their original score for Pascal VOC 2007.
Introduction
While Deep Convolutional Neural Networks (CNNs) have revolutionized computer vision, they are often treated as "black boxes." In transfer learning—where a model trained on one task (like ImageNet) is used as a feature extractor for another—it remains unclear how much of the generated data is strictly necessary. Large feature vectors are computationally expensive to store, transmit, and process, especially for mobile or edge computing.
This paper investigates the "resiliency" of deep learning features by putting them under "stress"—specifically, aggressively reducing their dimensionality and numerical precision. The authors discover that deep neural networks exhibit massive redundancy, allowing for a staggering 98.4% compression rate of feature vectors with less than a 1% drop in accuracy during transfer tasks.
Methods
The authors propose a "stress test" framework consisting of three main types of interference applied to feature vectors extracted from pre-trained models like VGG-M and GoogLeNet:
- Dimensionality Reduction (DR):
- DR-1 (Random): Dropping dimensions at random to test general robustness.
- DR-2 (PCA-based): Using Principal Component Analysis to discard dimensions with the least variance.
- Quantization (Q): Reducing the numerical precision of the data.
- Q-1 (Global): Applying the same quantization scale to all dimensions.
- Q-2 (Per-dimension): Adapting the scale for each individual dimension to better preserve its specific range.
- Feature Compression (FC): Combining PCA-based reduction and per-dimension quantization simultaneously.
After "stressing" the features, they are fed into a simple linear SVM to evaluate how well they still perform on a target classification task.
Experimental Setup
The study evaluates these stressors across three distinct datasets to ensure findings aren't task-specific:
- Pascal VOC 2007: A standard benchmark for object classification.
- MIT-67 Indoor: A dataset focused on indoor scene recognition.
- UPMC Food-101: A challenging dataset for food category classification.
Results are compared against a classical non-deep learning baseline: the BossaNova Bag-of-Words model.
Results and Discussion
The most striking insight is that deep networks produce far more data than is strictly necessary for classification. This redundancy acts as a buffer, making the features incredibly resilient to noise and data loss.
Key Findings:
- The Power of PCA: While random dropping (DR-1) eventually hurts performance, PCA-based reduction (DR-2) is highly effective. Discarding the vast majority of dimensions results in virtually no change in accuracy, proving that the "signal" is concentrated in a very small subspace.
- Adaptive Quantization: In quantization, the Q-2 method (adapting to each dimension) significantly outperformed the global Q-1 method. This indicates that while the precision of values can be low (down to 2 bits), the relative scale of each feature dimension is critical.
- Superiority over Classical Methods: Deep features were significantly more resilient to stress than the BossaNova baseline. Even when deep features were compressed to be 15 times smaller than the baseline's features, they still provided better accuracy.
- Massive Compression: On the Pascal VOC 2007 dataset, the authors achieved a 98.4% compression rate while losing only 0.88% of the original mAP (Mean Average Precision).
Conclusion
This work demonstrates that deep features are inherently robust and redundant. This opens the door for creating powerful, compact descriptors suitable for resource-constrained environments like mobile or edge devices. Future research could explore integrating these stressors directly into the training process to bake-in extreme efficiency from the start.
Summary updated