Understanding the Effects of Pre-Training for Object Detectors via Eigenspectrum

Yosuke Shinya, Edgar Simo-Serra, Taiji Suzuki

Abstract

ImageNet pre-training has been regarded as essential for training accurate object detectors for a long time. Recently, it has been shown that object detectors trained from randomly initialized weights can be on par with those fine-tuned from ImageNet pre-trained models. However, the effects of pre-training and the differences caused by pre-training are still not fully understood. In this paper, we analyze the eigenspectrum dynamics of the covariance matrix of each feature map in object detectors. Based on our analysis on ResNet-50, Faster R-CNN with FPN, and Mask R-CNN, we show that object detectors trained from ImageNet pre-trained models and those trained from scratch behave differently from each other even if both object detectors have similar accuracy. Furthermore, we propose a method for automatically determining the widths (the numbers of channels) of object detectors based on the eigenspectrum. We train Faster R-CNN with FPN from randomly initialized weights, and show that our method can reduce ~27% of the parameters of ResNet-50 without increasing Multiply-Accumulate operations and losing accuracy. Our results indicate that we should develop more appropriate methods for transferring knowledge from image classification to object detection (or other tasks).

Introduction

For years, the standard practice in computer vision involved fine-tuning a model pre-trained on ImageNet for downstream tasks like object detection. However, recent findings have challenged this paradigm, showing that models trained from scratch can achieve comparable accuracy if given sufficient training time. This raises a critical question: do these models converge to the same internal representations? This research investigates the "intrinsic dimensionality" of neural network layers to determine if pre-trained and scratch-trained models are fundamentally different under the hood.

Intrinsic Architecture Search (IAS)

To analyze and optimize network structures, we introduce Intrinsic Architecture Search (IAS). This framework leverages the eigenspectrum of the covariance matrix of feature maps to measure information redundancy. The process involves three main steps:

  1. Shrinking: New layer widths are determined by the number of eigenvalues exceeding a threshold $T$, effectively pruning redundant channels.
  2. Adjusting: The architecture is modified to satisfy structural constraints, such as residual shortcut connections.
  3. Expanding: The network is scaled back up using a uniform multiplier to meet specific resource targets, such as matching the original parameter count.
The Intrinsic Architecture Search (IAS) process applied to fine-tuned (I) and scratch-trained (S) models. The resulting "ResiaNet" architectures reveal significant structural divergence despite similar final accuracies.

Eigenspectrum Dynamics and Insights

Our analysis reveals several key phenomena regarding how models learn and adapt to detection tasks:

  • The "Forgetting" Phenomenon: In fine-tuned models, the final stages of the backbone (e.g., Stage 5 in ResNet) often become highly redundant. The model effectively "forgets" specific ImageNet features that are not required for the coarser localization tasks in detection.
  • The Rebound Effect: We observed that eigenvalues in certain layers initially decrease (simplification) and then "rebound" (increasing complexity) following the first learning rate decay, suggesting the model captures finer details as optimization stabilizes.
  • Structural Divergence: Fine-tuned models and scratch-trained models prefer different optimal widths. Fine-tuned models typically benefit from wider middle stages to preserve information flow from pre-training, whereas scratch-trained models develop different redundancy patterns.
Evolution of the number of significant eigenvalues over training iterations. The plot illustrates the "forgetting" effect in early stages and the "rebound" effect occurring after learning rate decays.

Results and Discussion

Applying IAS to a ResNet-50 backbone for Faster R-CNN on the MS COCO dataset yielded significant improvements in efficiency and performance:

  • Parameter Efficiency: We produced a version of ResNet-50 with ~27% fewer parameters (18.6M vs. 25.5M) that maintains the baseline Average Precision (AP).
  • Performance Gains: When constrained to the original parameter budget, the discovered ResiaNet architecture improved AP from 35.5 to 35.8.
  • Faster Convergence: The eigenspectrum-optimized architectures allowed scratch-trained models to converge more quickly than standard ResNet designs.

Conclusion

While pre-training and scratch-training can reach similar performance levels, their internal dynamics and optimal architectures differ significantly. By using the eigenspectrum to identify and remove redundancy, we can design more efficient detectors. Future work will explore extending this analysis to optimize network depth and input resolution, potentially integrating these insights into broader Neural Architecture Search (NAS) frameworks.

Summary updated

Abstract

ImageNet pre-training has been regarded as essential for training accurate object detectors for a long time. Recently, it has been shown that object detectors trained from randomly initialized weights can be on par with those fine-tuned from ImageNet pre-trained models. However, the effects of pre-training and the differences caused by pre-training are still not fully understood. In this paper, we analyze the eigenspectrum dynamics of the covariance matrix of each feature map in object detectors. Based on our analysis on ResNet-50, Faster R-CNN with FPN, and Mask R-CNN, we show that object detectors trained from ImageNet pre-trained models and those trained from scratch behave differently from each other even if both object detectors have similar accuracy. Furthermore, we propose a method for automatically determining the widths (the numbers of channels) of object detectors based on the eigenspectrum. We train Faster R-CNN with FPN from randomly initialized weights, and show that our method can reduce ~27% of the parameters of ResNet-50 without increasing Multiply-Accumulate operations and losing accuracy. Our results indicate that we should develop more appropriate methods for transferring knowledge from image classification to object detection (or other tasks).
BibTeX
@misc{shinya2019understanding,
  author = {Yosuke Shinya and Edgar Simo-Serra and Taiji Suzuki},
  title = {{Understanding the Effects of Pre-Training for Object Detectors via Eigenspectrum}},
  year = {2019},
  eprint = {1909.04021},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1909.04021},
}
Figure