TL;DR
This paper demonstrates how the Intel Neural Compute Stick 2 (NCS2) can dramatically accelerate deep learning tasks like image recognition and object tracking on a Raspberry Pi 4B. The study shows significant speedups (up to 1400%) and improved power efficiency, making real-time AI feasible on low-cost, embedded systems.
What Problem Does This Paper Address?
Deploying deep learning models on embedded systems, like those found in IoT devices or autonomous systems, is challenging. These systems often have limited processing power, memory, and strict power constraints. While powerful GPUs and cloud TPUs can handle complex neural networks, they are too power-hungry and expensive for many edge applications. This creates a need for efficient, low-power hardware accelerators that can boost the performance of embedded devices for real-time AI tasks such as image recognition and object tracking, without breaking the bank or draining the battery.
Methods
The researchers evaluated the Intel Neural Compute Stick 2 (NCS2) on a Raspberry Pi 4B. The NCS2 is a dedicated hardware accelerator (a VPU - Vision Processing Unit) designed to offload computationally intensive tasks from the CPU. They used the OpenVINO toolkit (version 2022.3.2) to run deep learning models.
The study focused on two main tasks:
- Image Recognition: They used a single MobileNetV3-Small model, a lightweight architecture suitable for embedded systems.
- Object Tracking: They employed a two-stage approach using a pre-trained person-detection model (MobileNetV2-based) and a person-reidentification model (OSNet-based) combined with the Deep SORT algorithm for tracking.
They measured inference speed (Frames Per Second - FPS) and power consumption for both the Raspberry Pi 4B's CPU alone and the Raspberry Pi 4B with the NCS2. Additionally, they investigated the impact of CPU clock frequency (overclocking and downclocking) on performance.
Datasets
The paper states that the test images for image recognition were "self-produced" and adjusted to match the model's input dimension (224x224 pixels). For object tracking, they used video streams from a USB camera. Specific public datasets were not explicitly mentioned for training or evaluation, but the models used (MobileNetV3-Small, MobileNetV2, OSNet) are standard architectures often trained on large, public datasets.
Key Insights and Ideas
- NCS2 is a Game-Changer for Edge AI: The most significant takeaway is that the NCS2 provides a substantial performance boost for deep learning on resource-constrained devices, making previously infeasible real-time applications possible.
- CPU Overclocking Has Limited Impact: While increasing the Raspberry Pi's CPU clock speed offers some improvement, it's marginal compared to the gains from using the NCS2. This highlights the inherent limitations of general-purpose CPUs for complex AI tasks.
- Power Efficiency is Crucial: The NCS2 not only speeds up processing but also does so more efficiently, consuming significantly less power per frame processed, which is vital for battery-powered or energy-sensitive applications.
- Reproducibility Matters: The authors emphasize the difficulty in setting up the OpenVINO environment for older NCS2 versions and provide a GitHub repository and detailed setup instructions to ensure their work is reproducible.
Key Results
- Image Recognition Speedup: The NCS2 increased image recognition performance by approximately 400%, achieving over 50 FPS.
- Object Tracking Speedup: The NCS2 accelerated real-time object tracking by 1200% to 1400%, reaching over 20 FPS.
- Power Efficiency: The NCS2 demonstrated 200% to 400% better power efficiency per image/frame compared to the CPU alone. For person tracking, it was up to four times more power-efficient per frame.
- CPU vs. NCS2: Even with CPU overclocking, the performance gains were minimal compared to the NCS2. The CPU alone achieved around 1.6 FPS for tracking, while the NCS2 reached ~21 FPS.
Limitations and Future Work
- OpenVINO Version Compatibility: The paper notes that OpenVINO versions released after March 2022 no longer support the NCS2, making setup challenging for users of newer software. They provide instructions for an older, compatible version (2022.3.2).
- Specific Models Tested: The reported speedups and efficiency gains are specific to the MobileNetV3-Small and MobileNetV2/OSNet models used. Performance might vary with different architectures.
- Reproducibility Challenges: The authors highlight the effort required to set up the development environment, suggesting that easier installation and better documentation for older hardware/software combinations would be beneficial.
Future work could explore the NCS2's performance with a wider range of DNN models and on other embedded platforms. Further investigation into optimizing the OpenVINO environment for broader compatibility would also be valuable.
Summary updated