A Proper Orthogonal Decomposition Approach for Parameters Reduction of Single Shot Detector Networks

Laura Meneghetti, Nicola Demo, Gianluigi Rozza

TL;DR

This paper introduces a novel method to reduce the computational complexity of Single Shot Detector (SSD) networks for object detection. By applying Proper Orthogonal Decomposition (POD) to compress the base network, they achieve significant reductions in model size and training time, albeit with a slight decrease in detection accuracy.

What Problem Does This Paper Address?

Deep neural networks, particularly those used for object detection like SSD, have become incredibly powerful but also computationally demanding. This poses a significant challenge for deployment on resource-constrained embedded systems, common in industrial applications. The sheer number of parameters in these networks leads to long training times and substantial storage requirements. While some approaches focus on using lighter backbone networks or pruning, this paper aims to tackle the problem by reducing the dimensionality of the existing, high-performing network architectures.

Methods

The core of this paper is the application of Proper Orthogonal Decomposition (POD), a classical model order reduction technique, to compress the "base net" of an object detection architecture. The process involves:

  1. Network Splitting: The base convolutional neural network (CNN) is split into two parts: a "pre-model" (layers before a chosen cut-off layer) and a "post-model" (layers after the cut-off).
  2. Dimensionality Reduction: The output of the pre-model, which is a high-dimensional feature representation, is then compressed using POD. This involves:
  3. Collecting the pre-model outputs from a training dataset into a matrix.
  4. Performing Singular Value Decomposition (SVD) on this matrix to obtain POD modes.
  5. Constructing a projection matrix using the top 'r' POD modes (where 'r' is the desired reduced dimension).
  6. Projecting the pre-model output onto this lower-dimensional space.
  7. Predictor Connection: The reduced feature representation is then fed into the original predictor block of the SSD network. This means the predictor now receives a compressed version of the features, rather than the full, high-dimensional output from the base net.

The key novelty lies in adapting POD, traditionally used for PDEs, to compress the feature space of deep neural networks for object detection.

Datasets

  • PASCAL Visual Object Classes (VOC) dataset: Used for evaluating the performance of the reduced network on a larger scale.
  • A reduced version of VOC2007: Specifically, a subset of 300 images categorized into "cats" and "dogs" was used for initial training and evaluation. This smaller dataset allowed for faster experimentation and demonstration of the core concepts.

Key Insights and Ideas

  • POD as a Feature Compressor: The paper demonstrates that POD can effectively capture the most important directions in the high-dimensional feature space of a CNN, allowing for significant compression without completely losing critical information.
  • Trade-off Between Accuracy and Efficiency: The core finding is that reducing the network's dimensionality via POD leads to substantial gains in storage space and training time, but at the cost of a decrease in mean Average Precision (mAP). This highlights a fundamental trade-off that needs to be managed.
  • Impact of Problem Complexity: The authors observe that the accuracy drop is more pronounced when dealing with a larger number of object classes (20 classes in PASCAL VOC vs. 2 classes in the cat-dog dataset). This suggests that the complexity of the detection task directly influences how much information can be discarded.
  • POD Mode Storage Can Be a Bottleneck: A significant limitation identified is that the storage required for the POD modes themselves can become substantial, especially when dealing with very high-dimensional pre-model outputs. This can, in some cases, negate the memory savings.

Key Results

  • Storage Reduction: Achieved a reduction of 15% in storage for the cat-dog dataset and 22% for the PASCAL VOC dataset.
  • Training Time Reduction: The training time was halved for the PASCAL VOC dataset.
  • Accuracy Impact:
    • On the cat-dog dataset, mAP decreased by 11% (from 70.2% to 59%).
  • On the PASCAL VOC dataset, mAP decreased by 38% (from 77.8% to 39%).
  • Qualitative Results: Figure 2 shows visual comparisons of object detection results, indicating that the reduced network can still detect objects, though with potentially less precision than the original.

Limitations and Future Work

  • Accuracy Degradation: The most significant limitation is the noticeable drop in detection accuracy, particularly for more complex datasets.
  • POD Mode Storage: The memory footprint of the POD modes themselves can be a limiting factor for compression, especially with high-dimensional features.
  • Manual Cut-off Layer Selection: The choice of the cut-off layer ('l') is currently empirical and not automated.
  • Future Work:
    • Investigating hyperreduction techniques or POD variants to further minimize the data that needs to be saved.
  • Developing an automatic method to determine the optimal cut-off layer based on the problem's complexity.
  • Exploring the trade-off between problem complexity (e.g., number of classes) and the degree of network compression.

Summary updated

Abstract

As a major breakthrough in artificial intelligence and deep learning, Convolutional Neural Networks have achieved an impressive success in solving many problems in several fields including computer vision and image processing. Real-time performance, robustness of algorithms and fast training processes remain open problems in these contexts. In addition object recognition and detection are challenging tasks for resource-constrained embedded systems, commonly used in the industrial sector. To overcome these issues, we propose a dimensionality reduction framework based on Proper Orthogonal Decomposition, a classical model order reduction technique, in order to gain a reduction in the number of hyperparameters of the net. We have applied such framework to SSD300 architecture using PASCAL VOC dataset, demonstrating a reduction of the network dimension and a remarkable speedup in the fine-tuning of the network in a transfer learning context.
BibTeX
@inproceedings{meneghetti2022proper,
  author = {Laura Meneghetti and Nicola Demo and Gianluigi Rozza},
  title = {{A Proper Orthogonal Decomposition Approach for Parameters Reduction of Single Shot Detector Networks}},
  booktitle = {ICIP},
  year = {2022},
  doi = {10.1109/icip46576.2022.9897513},
}
Figure