TL;DR
This paper provides a comprehensive benchmark of over 40 deep neural network (DNN) architectures for image recognition, evaluating them across various performance metrics like accuracy, complexity, memory usage, and inference time on two distinct hardware platforms. The findings offer crucial insights for practitioners and researchers to select optimal DNNs for resource-constrained applications and understand the trade-offs between different architectural choices.
What Problem Does This Paper Address?
The rapid advancement in deep neural networks (DNNs) for image recognition has led to a proliferation of new architectures. While many papers focus on improving accuracy, there's a significant gap in understanding how these architectures perform under real-world constraints, particularly concerning computational cost, memory footprint, and inference speed. Existing studies often focus on a limited number of models or a single hardware platform, making it difficult for practitioners to choose the best DNN for their specific deployment needs (e.g., embedded systems vs. high-end GPUs). This paper aims to bridge this gap by providing a systematic and comparative analysis of a wide range of state-of-the-art DNNs across multiple performance indicators and hardware setups.
Methods
The authors developed a benchmark framework using Python and PyTorch, leveraging cuDNN and CUDA for GPU acceleration. They evaluated over 40 representative DNN architectures, including well-known ones like AlexNet, VGG, ResNet, Inception, and newer models like MobileNet and ShuffleNet. The evaluation was conducted on two distinct hardware platforms: a high-end workstation with an NVIDIA Titan X Pascal GPU and an embedded system with an NVIDIA Jetson TX1 GPU. This dual-platform approach allows for a direct comparison of performance under vastly different computational capacities.
The following performance indices were measured:
- Accuracy Rate: Top-1 and Top-5 accuracy on the ImageNet-1k validation set, using center crop evaluation.
- Model Complexity: Total number of learnable parameters, measured in MB.
- Memory Usage: Total memory consumption, including model and batch processing memory, for various batch sizes (1 to 64).
- Computational Complexity: Floating-point operations (FLOPs) required for a single forward pass.
- Inference Time: Time taken per image for both GPUs, measured in milliseconds and converted to frames per second (FPS).
Datasets
The primary dataset used for evaluation was ImageNet-1k. The authors utilized pre-trained models on this dataset for their experiments. No new benchmarks were introduced; instead, they focused on evaluating existing models on a standardized task.
Key Insights and Ideas
- Accuracy vs. Operations: Recognition accuracy does not necessarily increase with the number of operations. Some architectures achieve high accuracy with relatively fewer operations.
- Model Complexity vs. Accuracy: There isn't a linear relationship between model complexity (number of parameters) and recognition accuracy. Some models with higher complexity perform similarly to those with lower complexity.
- Parameter Efficiency: Not all DNN models utilize their parameters with the same level of efficiency. Some models are more "parameter-efficient" than others.
- Throughput vs. Accuracy: The desired inference throughput (images per second) imposes an upper bound on achievable accuracy.
- Hardware Dependency: Most models can achieve super real-time performance on a high-end GPU (Titan Xp), but only a select few can do so on an embedded system (Jetson TX1).
- Minimum GPU Memory: Even DNNs with very low model complexity require a minimum GPU memory footprint of approximately 0.6GB.
- Model Complexity as Memory Predictor: Model complexity can reliably estimate the total memory utilization for a batch size of 1.
Key Results
- Top Performer: NASNet-A-Large achieved the highest Top-1 and Top-5 accuracy but also had the highest computational complexity.
- Efficient Models: Among models with low computational complexity (under 5 G-FLOPS), SE-ResNeXt-50 (32x4d) showed the highest accuracy with a low parameter count.
- Real-time Performance: On the Titan Xp, almost all models achieved super real-time performance. On the Jetson TX1, only a few models (e.g., SqueezeNets, MobileNets, ResNet-18) could achieve super real-time performance with a batch size of 1.
- Memory Usage: For a batch size of 1 on the Titan Xp, most models required less than 1GB of memory, with some exceptions like NASNet-A-Large and SENet-154.
- Best DNNs under Constraints: The paper provides tables (Table 3) identifying the best DNNs for specific memory and inference time constraints on both the Titan Xp and Jetson TX1, highlighting trade-offs between accuracy and resource limitations. For instance, on the Titan Xp with a low memory constraint and high speed, DPN-68 achieved 75.95% accuracy.
Limitations and Future Work
- Limited Batch Sizes for Jetson TX1: Missing data in Table 1 for the Jetson TX1 indicates a lack of sufficient system memory to process larger batch sizes for some models.
- Future Work: The authors plan to add interactive plots to their publicly available repository to allow researchers to explore results more dynamically and add new entries to the benchmark. They also aim to make the analysis more comprehensive by including additional metrics and models.
Summary updated