Searching Effective Network Parameters to Construct Convolutional Neural Networks for Object Detection

Nuri Kim, Dong‐Hoon Lee, Songhwai OhJournal of KIISE 2017 · DOI

TL;DR

This paper investigates the impact of key hyperparameters—dropout ratio, anchor box size, and activation function—on the performance of Faster R-CNN for object detection. The study finds that a dropout ratio of 0.3 yields optimal results, anchor box size has minimal impact, and leaky ReLU with a specific negative slope performs well, offering practical insights for optimizing deep learning models in computer vision.

What Problem Does This Paper Address?

Deep learning has revolutionized fields like image recognition and object detection, achieving remarkable performance. However, a deep understanding of how these complex neural networks operate and why certain configurations perform better remains elusive. This paper aims to bridge this gap by empirically analyzing the influence of critical hyperparameters within the Faster R-CNN architecture on object detection accuracy. Specifically, it seeks to understand the role of dropout probability in preventing overfitting, the significance of anchor box dimensions for detecting objects of various scales, and the effect of different activation functions on learning.

Methods

The researchers employed the Faster R-CNN framework as their baseline model. They systematically varied three key hyperparameters to observe their impact on object detection performance:

  • Dropout Ratio: They experimented with dropout probabilities ranging from 0.1 to 0.9, with increments of 0.2 (0.1, 0.3, 0.5, 0.7, 0.9). Dropout is a regularization technique where randomly selected neurons are ignored during training to prevent overfitting.
  • Anchor Box Size: The study analyzed three different sets of anchor box sizes: (64², 128², 256²), (128², 256², 512²), and (16², 32², 64²). Anchor boxes are pre-defined bounding boxes of various scales and aspect ratios used to predict object locations.
  • Activation Function: The paper focused on the leaky ReLU activation function, varying the slope of its negative domain (0, 0.01, 0.02). Leaky ReLU is a variant of ReLU that allows a small, non-zero gradient when the input is negative, aiming to prevent "dying ReLU" problems.

Additionally, they compared the effectiveness of dropout with batch normalization, another common regularization technique that normalizes the inputs to a layer.

Datasets

The experiments were conducted using the PASCAL VOC 2007 dataset. This dataset is commonly used for object detection tasks and contains 10,000 images annotated with bounding boxes for 20 object classes. The models were pre-trained on the ILSVRC2012 dataset.

Key Insights and Ideas

  • Dropout is Crucial for Regularization: The study strongly suggests that dropout plays a significant role in improving object detection performance by preventing overfitting. The optimal dropout ratio found (0.3) indicates a sweet spot between underfitting (too little dropout) and overfitting (too much dropout).
  • Anchor Box Size is Less Critical Than Expected: Surprisingly, the size of the anchor boxes did not show a strong correlation with the final object detection performance. This suggests that the network can learn to adapt to object scales effectively, even with anchor boxes that don't perfectly match the ground truth sizes, as long as there's a reasonable distribution.
  • Dropout and Batch Normalization are Complementary, Not Substitutes: The research found that batch normalization alone did not achieve the same performance as dropout. This implies that these two regularization techniques address different aspects of network training and might be best used in conjunction rather than as replacements for each other.
  • Leaky ReLU's Negative Slope Matters: The study highlights that a small, non-zero negative slope in leaky ReLU (specifically 0.02) can lead to better performance compared to a standard ReLU (slope of 0) or even a slightly smaller slope (0.01). This suggests that allowing some gradient flow for negative inputs can be beneficial.

Key Results

  • Optimal Dropout Ratio: A dropout ratio of 0.3 yielded the best overall performance (mAP).
  • Anchor Box Insensitivity: The performance differences between the tested anchor box configurations (Models A, B, and C) were relatively minor, indicating that anchor box size is not a primary driver of performance in this context.
  • Batch Normalization vs. Dropout: Replacing dropout with batch normalization resulted in a significant performance drop (around 20 mAP), underscoring dropout's importance.
  • Leaky ReLU Performance: Leaky ReLU with a negative slope of 0.02 achieved comparable or slightly better performance than other tested slopes.

Limitations and Future Work

The authors acknowledge that their study focused on varying one hyperparameter at a time while keeping others fixed. This approach does not account for potential trade-offs between different hyperparameters. Future work should explore the interactions and combined effects of these parameters to achieve a more holistic understanding and potentially further optimize network design. Additionally, the paper suggests investigating other factors that influence network performance, such as optimization techniques and network architecture variations.

Summary updated

Abstract

최근 몇 년간 딥러닝(deep learning)은 음성 인식, 영상 인식, 물체 검출을 비롯한 다양한 패턴인식 분야에서 혁신적인 성능 발전을 거듭해왔다. 그에 비해 네트워크가 어떻게 작동하는지에 대한 깊은 이해는 잘 이루어지지 않고 있다. 본 논문은 효과적인 신경망 네트워크를 구성하기 위해 네트워크 파라미터들이 신경망 내부에서 어떻게 작동하고, 어떤 역할을 하고 있는지 분석하였다. Faster R-CNN 네트워크를 기반으로 하여 신경망의 과적합(overfitting)을 막는 드랍아웃(dropout) 확률과 앵커 박스 크기, 그리고 활성 함수를 변화시켜 학습한 후 그 결과를 분석하였다. 또한 드랍아웃과 배치 정규화(batch normalization) 방식을 비교해보았다. 드랍아웃 확률은 0.3일 때 가장 좋은 성능을 보였으며 앵커 박스의 크기는 최종 물체 검출 성능과 큰 관련이 없다는 것을 알 수 있었다. 드랍아웃과 배치 정규화 방식은 서로를 완전히 대체할 수는 없는 것을 확인할 수 있었다. 활성화 함수는 음수 도메인의 기울기가 0.02인 leaky ReLU가 비교적 좋은 성능을 보였다.
BibTeX
@article{kim2017searching,
  author = {Nuri Kim and Dong‐Hoon Lee and Songhwai Oh},
  title = {{Searching Effective Network Parameters to Construct Convolutional Neural Networks for Object Detection}},
  journal = {Journal of KIISE},
  year = {2017},
  doi = {10.5626/jok.2017.44.7.668},
}
Figure