Abstract
Feature extraction is a critical component of computer vision, with modern algorithms incorporating Deep Neural Networks (DNN) as feature extractors to create Deep Convolutional Activation Features (DeCAF). This study analyzes the general discriminability of novel object visual appearances encoded into the DeCAF space of six leading DNN architectures: VGG19, InceptionV3, ResNet50, DenseNet121, MobileNet, and NASNetLarge. By characterizing Mahalanobis distances and cosine similarities across the VOT2015 and UAV123 benchmark datasets, the study finds that different architectures lead to distinct network feature focuses. These results provide a framework for efficiently comparing estimated network performance for any labeled visual dataset.
Introduction
Transfer learning allows researchers to leverage models trained on massive datasets like ImageNet for novel object recognition tasks with limited data. However, pre-trained models are often treated as "black boxes," with little understanding of which architecture provides the most effective feature space for specific novel classes. This is particularly vital in long-term object tracking, where a target must be consistently distinguished from a dynamic background. This paper introduces a quantitative framework to evaluate how well these networks "see" and separate new, unseen objects.
Methodology
The study analyzes six prominent architectures by extracting features from their deepest convolutional layers. These features are modeled as "manifolds"—statistical clusters in high-dimensional space. Two primary metrics are used for evaluation:
- Mahalanobis Distance: Measures the separation between clusters while accounting for internal variance and shape.
- Cosine Similarity: Measures the angular overlap between feature vectors, indicating perceived similarity between classes.
The analysis utilized the VOT2015 and UAV123 benchmarks, covering 294 novel classes, including both target objects (TG) and backgrounds (BG). The researchers also examined the impact of sample size (10 vs. 100 vs. full sequences) and bounding box noise (3-pixel jitter).
Architectural Personalities and Feature Focus
The results reveal that different networks possess distinct "representational personalities":
- ResNet as a Background Specialist: ResNet50 showed uniquely high discriminability between different background types, making it ideal for scenes with complex environmental distractors.
- NASNet as a Target Specialist: NASNetLarge excelled at separating actual objects of interest (target-to-target discrimination).
- MobileNet’s Distinctiveness: Despite its efficiency-focused design, MobileNet produced the lowest cosine similarities, creating highly distinct, non-overlapping representations that minimize confusion between similar objects.
Key Results and Quantitative Analysis
The quantitative comparison, visualized through similarity and distance matrices, highlights the trade-offs between architectures. NASNet produced the largest Mahalanobis distances for target-to-target (TG-TG) separation, while ResNet achieved superior target-to-background (TG-BG) separation.
Furthermore, the study identified a "Sample Size Illusion": Mahalanobis distances were significantly higher when calculated from only 10 frames compared to full sequences. This suggests that small datasets can provide a misleadingly optimistic view of object separability by failing to capture the full range of appearance variations. Conversely, the features proved remarkably robust to "jitter," with 3-pixel bounding box errors having negligible impact on discriminability.
Conclusion
This study demonstrates that the choice of DNN backbone significantly impacts the discriminability of novel objects in feature space. While this analysis focused on the deepest convolutional layers, future work could explore earlier layers for tasks requiring higher spatial resolution. The proposed methodology provides a robust tool for informed network selection in custom computer vision applications, moving beyond the "black box" approach to transfer learning.
Summary updated