Deep Neural Networks Under Stress

Micael Carvalho, Matthieu Cord, Sandra Eliza Fontes de Avila, Nicolas Thome, Eduardo Valle

TL;DR

This paper investigates the "resiliency" of deep learning features by putting them under "stress"—specifically, aggressively reducing their dimensionality and numerical precision. The authors discover that deep neural networks exhibit massive redundancy, allowing for a staggering 98.4% compression rate of feature vectors with less than a 1% drop in accuracy during transfer learning tasks.

What Problem Does This Paper Address?

While Deep Convolutional Neural Networks (CNNs) have revolutionized computer vision, they are often treated as "black boxes." We know they work, but we don't always understand the nature of the representations they learn. Specifically, in transfer learning—where a model trained on one task (like ImageNet) is used as a feature extractor for another (like identifying food)—it’s unclear how much of the generated data is actually necessary.

Large feature vectors are computationally expensive to store, transmit, and process, especially for mobile or edge computing. This paper asks: How much information can we "destroy" before the model's performance collapses? By answering this, the authors shed light on the inherent redundancy and robustness of deep representations compared to classical computer vision techniques.

Methods

The authors propose a "stress test" framework consisting of three main types of interference applied to feature vectors extracted from pre-trained models (primarily VGG-M and GoogLeNet):

  1. Dimensionality Reduction (DR):
    • DR-1 (Random): Dropping dimensions at random to test general robustness.
  2. DR-2 (PCA-based): Using Principal Component Analysis to discard dimensions with the least variance.
  3. Quantization (Q): Reducing the numerical precision of the data.
  4. Q-1 (Global): Applying the same quantization scale to all dimensions.
  5. Q-2 (Per-dimension): Adapting the scale for each individual dimension to better preserve its specific range.
  6. Feature Compression (FC): Combining PCA-based reduction and per-dimension quantization simultaneously.

After "stressing" the features, they are fed into a simple linear SVM to evaluate how well they still perform on a target classification task.

Datasets

The study evaluates these stressors across three distinct datasets to ensure the findings aren't task-specific:

  • Pascal VOC 2007: A standard benchmark for object classification (20 classes).
  • MIT-67 Indoor: A dataset focused on indoor scene recognition (67 classes).
  • UPMC Food-101: A challenging dataset for food category classification (101 classes).

They also compare their results against a classical non-deep learning baseline: the BossaNova Bag-of-Words model.

Key Insights and Ideas

  • Deep Features are Inherently Redundant: The most striking insight is that deep networks produce far more data than is strictly necessary for classification. This redundancy acts as a buffer, making the features incredibly resilient to noise and data loss.
  • GoogLeNet vs. VGG-M: GoogLeNet proved more robust to random dimensionality stripping than VGG-M. However, when comparing features of similar final sizes, the two models performed comparably, suggesting a fundamental limit to how much information is needed for these tasks.
  • The "Aha" Moment for PCA: While random dropping (DR-1) eventually hurts performance, PCA-based reduction (DR-2) is almost "magical." You can discard the vast majority of dimensions and see virtually no change in accuracy, proving that the "signal" is concentrated in a very small subspace.
  • Adaptivity Matters: In quantization, the Q-2 method (adapting to each dimension) significantly outperformed the global Q-1 method. This indicates that while the precision of the values can be low, the relative scale of each feature dimension is critical.

Key Results

  • Massive Compression: On the Pascal VOC 2007 dataset, the authors achieved a 98.4% compression rate while losing only 0.88% of the original mAP (Mean Average Precision).
  • Precision vs. Performance: For VGG-M on VOC 2007, they found they could reduce the feature values to just 2 bits (4 values) per dimension and still maintain nearly "vanilla" performance levels.
  • Superiority over Classical Methods: Deep features were significantly more resilient to stress than the BossaNova (Bag-of-Words) baseline. Even when deep features were compressed to be 15 times smaller than the baseline's features, they still provided better accuracy.

Limitations and Future Work

  • Transfer Learning Focus: The study focuses on using CNNs as fixed feature extractors (the "off-the-shelf" approach). It does not explore how "stress" affects end-to-end fine-tuning, where the network weights themselves are updated.
  • Linear Classifiers: The evaluation relies on linear SVMs. While standard for transfer learning, it's possible that non-linear classifiers might interact differently with "stressed" features.
  • Future Directions: The authors suggest this work opens the door for creating "powerful compact descriptors." Future research could look into integrating these "stressors" directly into the training process (e.g., via specialized loss functions) to bake-in extreme efficiency from the start.

Summary updated

Abstract

In recent years, deep architectures have been used for transfer learning with state-of-the-art performance in many datasets. The properties of their features remain, however, largely unstudied under the transfer perspective. In this work, we present an extensive analysis of the resiliency of feature vectors extracted from deep models, with special focus on the trade-off between performance and compression rate. By introducing perturbations to image descriptions extracted from a deep convolutional neural network, we change their precision and number of dimensions, measuring how it affects the final score. We show that deep features are more robust to these disturbances when compared to classical approaches, achieving a compression rate of 98.4%, while losing only 0.88% of their original score for Pascal VOC 2007.
BibTeX
@inproceedings{carvalho2016deep,
  author = {Micael Carvalho and Matthieu Cord and Sandra Eliza Fontes de Avila and Nicolas Thome and Eduardo Valle},
  title = {{Deep Neural Networks Under Stress}},
  booktitle = {ICIP},
  year = {2016},
  url = {https://openreview.net/forum?id=21efB8eGctY},
}

Loading related content...

Figure