Real-Time Energy Efficient Hand Pose Estimation: A Case Study

Mhd Rashed Al Koutayni, Vladimir Rybalkin, Jameel Malik, Ahmed Elhayek, Christian Weis, Gerd Reis, Norbert Wehn, Didier StrickerSensors 2020 · OpenReview

TL;DR

This paper tackles the challenge of deploying real-time 3D hand pose estimation on power-constrained, portable devices. By compressing a state-of-the-art Convolutional Neural Network (CNN) through quantization and implementing it on an FPGA, the authors achieve a significant speedup and energy efficiency improvement over GPU-based solutions, making advanced human-computer interaction applications more feasible.

What Problem Does This Paper Address?

Real-time 3D hand pose estimation is crucial for many emerging applications like virtual reality, in-air signature verification, and robotics. Current leading methods rely on deep Convolutional Neural Networks (CNNs), which are computationally intensive and typically run on powerful GPUs. However, GPUs are power-hungry and bulky, making them unsuitable for portable devices where low power consumption and small form factors are paramount. This creates a gap between the capabilities of advanced hand pose estimation and the requirements of practical, mobile applications.

Methods

The authors address this challenge in two main stages:

  1. Model Compression: They employ Quantization-Aware Training (QAT) to reduce the size and computational demands of the CNN. Instead of simply quantizing a pre-trained model, QAT retrains the network from scratch with constrained bitwidths for weights and activations. This process aims to achieve significant compression with minimal accuracy loss. They also utilize depthwise separable convolutions to further reduce computation.
  2. Hardware Acceleration: The compressed CNN is then implemented on a Field-Programmable Gate Array (FPGA), specifically a Xilinx UltraScale+ MPSoC. FPGAs are chosen for their inherent energy efficiency and ability to perform parallel computations, making them ideal for embedded systems. They design a hardware streaming architecture (HSA) where each CNN layer is mapped to a dedicated hardware block, connected in a pipeline. High-Level Synthesis (HLS) is used to generate the hardware description from a C++ model.

Datasets

The paper uses the NYU hand pose dataset for training and testing the CNN. This dataset provides RGBD images along with ground-truth 3D joint coordinates for hand poses.

Key Insights and Ideas

  • Quantization-Aware Training is Key: Simply quantizing a trained model isn't enough. Retraining with quantization constraints (QAT) is crucial for maintaining accuracy while achieving significant model compression.
  • FPGA for Efficiency: FPGAs offer a compelling alternative to GPUs for real-time, embedded applications requiring low power consumption. Their parallel processing capabilities are well-suited for CNN inference.
  • Hardware Streaming Architecture: Mapping CNN layers to dedicated hardware blocks and pipelining them allows for efficient data flow and high throughput, essential for real-time performance.
  • Trade-off Management: The work highlights the successful trade-off between model size/energy consumption and accuracy. A slight decrease in accuracy (3%) is accepted for substantial gains in speed and energy efficiency.

Key Results

The implemented FPGA solution demonstrates significant improvements over the original GPU implementation:

  • Speed: The FPGA implementation is 4.2x faster than the NVIDIA GeForce GTX 1070 GPU for a single input image (batch size = 1).
  • Energy Efficiency: The FPGA solution is 577.3x more energy efficient than the GPU implementation, consuming only 0.6676 mJ per image compared to 385.41 mJ.
  • Model Size: The quantized CNN model is 5.3x smaller than the original full-precision CNN.
  • Accuracy: The quantized CNN exhibits only a 3% decrease in accuracy compared to the full-precision model, with an average joint error of 17.75 mm.
  • Comparison with Embedded Platforms: The FPGA solution is significantly faster and more energy-efficient than other embedded platforms like NVIDIA Jetson Xavier and Raspberry Pi 3B+ for single-image inference.

Limitations and Future Work

  • Limitations:
    • The paper focuses on a specific CNN architecture and does not explore more complex, potentially higher-accuracy models.
  • While the FPGA solution is faster for single-image inference, GPUs show better performance for larger batch sizes, which is more relevant for training.
  • The authors acknowledge that the FPGA implementation is a simplified version of existing streaming architectures.
  • Future Work:
    • Explore other CNN topologies and deeper architectures for potentially higher accuracy.
  • Investigate further quantization techniques, possibly leading to binarized networks.
  • Experiment with different scanning methods for input streaming to reduce bottlenecks.
  • Incorporate advanced architectural features like skip and merge connections (e.g., ResNet, DenseNet) to improve accuracy.

Summary updated

Abstract

The estimation of human hand pose has become the basis for many vital applications where the user depends mainly on the hand pose as a system input. Virtual reality (VR) headset, shadow dexterous hand and in-air signature verification are a few examples of applications that require to track the hand movements in real-time. The state-of-the-art 3D hand pose estimation methods are based on the Convolutional Neural Network (CNN). These methods are implemented on Graphics Processing Units (GPUs) mainly due to their extensive computational requirements. However, GPUs are not suitable for the practical application scenarios, where the low power consumption is crucial. Furthermore, the difficulty of embedding a bulky GPU into a small device prevents the portability of such applications on mobile devices. The goal of this work is to provide an energy efficient solution for an existing depth camera based hand pose estimation algorithm. First, we compress the deep neural network model by applying the dynamic quantization techniques on different layers to achieve maximum compression without compromising accuracy. Afterwards, we design a custom hardware architecture. For our device we selected the FPGA as a target platform because FPGAs provide high energy efficiency and can be integrated in portable devices. Our solution implemented on Xilinx UltraScale+ MPSoC FPGA is 4.2× faster and 577.3× more energy efficient than the original implementation of the hand pose estimation algorithm on NVIDIA GeForce GTX 1070.
BibTeX
@inproceedings{koutayni2020real,
  author = {Mhd Rashed Al Koutayni and Vladimir Rybalkin and Jameel Malik and Ahmed Elhayek and Christian Weis and Gerd Reis and Norbert Wehn and Didier Stricker},
  title = {{Real-Time Energy Efficient Hand Pose Estimation: A Case Study}},
  booktitle = {Sensors},
  year = {2020},
  url = {https://openreview.net/forum?id=e4irwXj1sde},
}
Figure