TL;DR
This paper tackles the challenge of deploying real-time 3D hand pose estimation on power-constrained, portable devices. By compressing a state-of-the-art Convolutional Neural Network (CNN) through quantization and implementing it on an FPGA, the authors achieve a significant speedup and energy efficiency improvement over GPU-based solutions, making advanced human-computer interaction applications more feasible.
What Problem Does This Paper Address?
Real-time 3D hand pose estimation is crucial for many emerging applications like virtual reality, in-air signature verification, and robotics. Current leading methods rely on deep Convolutional Neural Networks (CNNs), which are computationally intensive and typically run on powerful GPUs. However, GPUs are power-hungry and bulky, making them unsuitable for portable devices where low power consumption and small form factors are paramount. This creates a gap between the capabilities of advanced hand pose estimation and the requirements of practical, mobile applications.
Methods
The authors address this challenge in two main stages:
- Model Compression: They employ Quantization-Aware Training (QAT) to reduce the size and computational demands of the CNN. Instead of simply quantizing a pre-trained model, QAT retrains the network from scratch with constrained bitwidths for weights and activations. This process aims to achieve significant compression with minimal accuracy loss. They also utilize depthwise separable convolutions to further reduce computation.
- Hardware Acceleration: The compressed CNN is then implemented on a Field-Programmable Gate Array (FPGA), specifically a Xilinx UltraScale+ MPSoC. FPGAs are chosen for their inherent energy efficiency and ability to perform parallel computations, making them ideal for embedded systems. They design a hardware streaming architecture (HSA) where each CNN layer is mapped to a dedicated hardware block, connected in a pipeline. High-Level Synthesis (HLS) is used to generate the hardware description from a C++ model.
Datasets
The paper uses the NYU hand pose dataset for training and testing the CNN. This dataset provides RGBD images along with ground-truth 3D joint coordinates for hand poses.
Key Insights and Ideas
- Quantization-Aware Training is Key: Simply quantizing a trained model isn't enough. Retraining with quantization constraints (QAT) is crucial for maintaining accuracy while achieving significant model compression.
- FPGA for Efficiency: FPGAs offer a compelling alternative to GPUs for real-time, embedded applications requiring low power consumption. Their parallel processing capabilities are well-suited for CNN inference.
- Hardware Streaming Architecture: Mapping CNN layers to dedicated hardware blocks and pipelining them allows for efficient data flow and high throughput, essential for real-time performance.
- Trade-off Management: The work highlights the successful trade-off between model size/energy consumption and accuracy. A slight decrease in accuracy (3%) is accepted for substantial gains in speed and energy efficiency.
Key Results
The implemented FPGA solution demonstrates significant improvements over the original GPU implementation:
- Speed: The FPGA implementation is 4.2x faster than the NVIDIA GeForce GTX 1070 GPU for a single input image (batch size = 1).
- Energy Efficiency: The FPGA solution is 577.3x more energy efficient than the GPU implementation, consuming only 0.6676 mJ per image compared to 385.41 mJ.
- Model Size: The quantized CNN model is 5.3x smaller than the original full-precision CNN.
- Accuracy: The quantized CNN exhibits only a 3% decrease in accuracy compared to the full-precision model, with an average joint error of 17.75 mm.
- Comparison with Embedded Platforms: The FPGA solution is significantly faster and more energy-efficient than other embedded platforms like NVIDIA Jetson Xavier and Raspberry Pi 3B+ for single-image inference.
Limitations and Future Work
- Limitations:
- The paper focuses on a specific CNN architecture and does not explore more complex, potentially higher-accuracy models.
- While the FPGA solution is faster for single-image inference, GPUs show better performance for larger batch sizes, which is more relevant for training.
- The authors acknowledge that the FPGA implementation is a simplified version of existing streaming architectures.
- Future Work:
- Explore other CNN topologies and deeper architectures for potentially higher accuracy.
- Investigate further quantization techniques, possibly leading to binarized networks.
- Experiment with different scanning methods for input streaming to reduce bottlenecks.
- Incorporate advanced architectural features like skip and merge connections (e.g., ResNet, DenseNet) to improve accuracy.
Summary updated