Abstract
Performance on high-end desktop GPUs is a poor predictor of how a model will behave on the "edge." This paper optimizes the state-of-the-art RefineDet object detector for embedded AI platforms, achieving real-time performance (~20 FPS) by identifying and mitigating hidden CPU bottlenecks in post-processing.
Introduction
In the world of deep learning, "fast" is relative. A model that tops the leaderboards on an NVIDIA Titan XP might crawl on a Jetson Xavier. This discrepancy exists because high-end workstations have massive CPU and memory resources that mask certain algorithmic inefficiencies.
The authors address the challenge of deploying RefineDet—a high-accuracy, "one-stage-meets-two-stage" object detector—onto edge devices like the NVIDIA Drive PX2 and Jetson Xavier. These platforms are common in autonomous driving and robotics but have much tighter constraints on power, heat, and the balance between CPU and GPU power. The goal is to find the "sweet spot" where accuracy remains high but inference happens in real-time.
Methods
The researchers took a multi-pronged approach to optimizing RefineDet:
- Backbone Architecture Search: They swapped the standard heavy backbones (like VGG-16 or ResNet-101) for a variety of lightweight alternatives, including ResNet-18, MobileNetV1/V2, Xception, and ResNeXt-26.
- The "rRefineDet" Variant: They introduced a "reduced" version of the architecture. By cutting the channel depth of the Transfer Connection Blocks (TCB) and intermediate layers from 256 to 128, they significantly reduced the computational load of the detector "head."
- NMS Parameter Tuning: They identified that Non-Maximum Suppression (NMS)—the algorithm that filters out redundant overlapping boxes—is a major bottleneck on edge devices because it typically runs on the CPU. They tuned the maximum number of candidate boxes (reducing it from 1000 to 400) and increased the confidence threshold to 0.1 to lighten the CPU's load.
The study exclusively used the MS-COCO 2017 object detection dataset for training and evaluation. Performance was measured using Mean Average Precision (mAP) on the val17 and test-dev17 sets.
Key Insights: The Hidden CPU Bottleneck
On high-end systems, the CPU is so fast that NMS feels "free." On edge platforms, the CPU is often the primary bottleneck. Even if the GPU finishes the neural network forward pass quickly, the system hangs while the CPU sorts through thousands of bounding boxes.
Furthermore, the "fastest" model changes depending on the hardware. For example, VGG-16 was the fastest architecture on the Titan XP, but ResNet-18 took the crown on the Drive PX2. This proves that developers cannot rely on desktop benchmarks when designing for the edge. Edge boards are far more sensitive to hyperparameter choices (like NMS settings) than desktop GPUs. A small tweak that barely affects speed on a Titan XP can result in a 2x speedup on an embedded board.
Key Results
The authors successfully reached ~20 FPS on both the Drive PX2 and Jetson Xavier. The rRefineDet320 with a ResNet-18 backbone emerged as a winner for edge use, hitting 21.6 FPS on the Drive PX2 with a respectable mAP of 33.5.
By simply adjusting NMS parameters (reducing candidates from 1000 to 400), they nearly doubled the frame rate on the Drive PX2 in several experiments with negligible loss in accuracy. While the original RefineDet on a Titan XP hits ~40 FPS, it drops to unusable levels on edge boards without the modifications proposed in this paper.
Limitations and Future Work
The authors acknowledge that while they achieved real-time speeds, there is still a trade-off between accuracy and velocity. Specifically:
- Head vs. Backbone: More detailed ablation studies are needed to understand the exact relationship between the backbone layers and the detector head's structure.
- Precision Optimization: The current study did not deeply explore FP16 (half-precision) or INT8 quantization using tools like NVIDIA TensorRT, which could potentially double the speed again on supported hardware.
- Platform Scope: The study is limited to NVIDIA's ecosystem; performance on other edge AI chips remains an open question.
Summary updated