Network Comparison Study of Deep Activation Feature Discriminability with Novel Objects

Michael Karnes, Alper YilmazarXiv 2022 · OpenReview

Abstract

Feature extraction is a critical component of computer vision, with modern algorithms incorporating Deep Neural Networks (DNN) as feature extractors to create Deep Convolutional Activation Features (DeCAF). This study analyzes the general discriminability of novel object visual appearances encoded into the DeCAF space of six leading DNN architectures: VGG19, InceptionV3, ResNet50, DenseNet121, MobileNet, and NASNetLarge. By characterizing Mahalanobis distances and cosine similarities across the VOT2015 and UAV123 benchmark datasets, the study finds that different architectures lead to distinct network feature focuses. These results provide a framework for efficiently comparing estimated network performance for any labeled visual dataset.

Introduction

Transfer learning allows researchers to leverage models trained on massive datasets like ImageNet for novel object recognition tasks with limited data. However, pre-trained models are often treated as "black boxes," with little understanding of which architecture provides the most effective feature space for specific novel classes. This is particularly vital in long-term object tracking, where a target must be consistently distinguished from a dynamic background. This paper introduces a quantitative framework to evaluate how well these networks "see" and separate new, unseen objects.

Methodology

The study analyzes six prominent architectures by extracting features from their deepest convolutional layers. These features are modeled as "manifolds"—statistical clusters in high-dimensional space. Two primary metrics are used for evaluation:

  1. Mahalanobis Distance: Measures the separation between clusters while accounting for internal variance and shape.
  2. Cosine Similarity: Measures the angular overlap between feature vectors, indicating perceived similarity between classes.

The analysis utilized the VOT2015 and UAV123 benchmarks, covering 294 novel classes, including both target objects (TG) and backgrounds (BG). The researchers also examined the impact of sample size (10 vs. 100 vs. full sequences) and bounding box noise (3-pixel jitter).

Architectural Personalities and Feature Focus

The results reveal that different networks possess distinct "representational personalities":

  • ResNet as a Background Specialist: ResNet50 showed uniquely high discriminability between different background types, making it ideal for scenes with complex environmental distractors.
  • NASNet as a Target Specialist: NASNetLarge excelled at separating actual objects of interest (target-to-target discrimination).
  • MobileNet’s Distinctiveness: Despite its efficiency-focused design, MobileNet produced the lowest cosine similarities, creating highly distinct, non-overlapping representations that minimize confusion between similar objects.

Key Results and Quantitative Analysis

The quantitative comparison, visualized through similarity and distance matrices, highlights the trade-offs between architectures. NASNet produced the largest Mahalanobis distances for target-to-target (TG-TG) separation, while ResNet achieved superior target-to-background (TG-BG) separation.

Sequence Pooled Cosine Similarity (top) and Log Mahalanobis Distance (bottom) matrices across six DNN architectures. Darker areas in the top plot and warmer colors in the bottom plot indicate better separation between classes.

Furthermore, the study identified a "Sample Size Illusion": Mahalanobis distances were significantly higher when calculated from only 10 frames compared to full sequences. This suggests that small datasets can provide a misleadingly optimistic view of object separability by failing to capture the full range of appearance variations. Conversely, the features proved remarkably robust to "jitter," with 3-pixel bounding box errors having negligible impact on discriminability.

Conclusion

This study demonstrates that the choice of DNN backbone significantly impacts the discriminability of novel objects in feature space. While this analysis focused on the deepest convolutional layers, future work could explore earlier layers for tasks requiring higher spatial resolution. The proposed methodology provides a robust tool for informed network selection in custom computer vision applications, moving beyond the "black box" approach to transfer learning.

Summary updated

Abstract

Feature extraction has always been a critical component of the computer vision field. More recently, state-of-the-art computer visions algorithms have incorporated Deep Neural Networks (DNN) in feature extracting roles, creating Deep Convolutional Activation Features (DeCAF). The transferability of DNN knowledge domains has enabled the wide use of pretrained DNN feature extraction for applications with novel object classes, especially those with limited training data. This study analyzes the general discriminability of novel object visual appearances encoded into the DeCAF space of six of the leading visual recognition DNN architectures. The results of this study characterize the Mahalanobis distances and cosine similarities between DeCAF object manifolds across two visual object tracking benchmark data sets. The backgrounds surrounding each object are also included as an object classes in the manifold analysis, providing a wider range of novel classes. This study found that different network architectures led to different network feature focuses that must to be considered in the network selection process. These results are generated from the VOT2015 and UAV123 benchmark data sets; however, the proposed methods can be applied to efficiently compare estimated network performance characteristics for any labeled visual data set.
BibTeX
@misc{karnes2022network,
  author = {Michael Karnes and Alper Yilmaz},
  title = {{Network Comparison Study of Deep Activation Feature Discriminability with Novel Objects}},
  year = {2022},
  eprint = {2202.03695},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2202.03695},
}
Figure