Real-time Multiple People Hand Localization in 4D Point Clouds

Hao Jiang, Quanzeng YouarXiv 2019 · arXiv

Introduction

Localizing hands in crowded and cluttered environments is a fundamental challenge for human-computer interaction and gesture recognition. Traditional methods typically rely on multi-view triangulation—which often fails during self-occlusion—or direct 3D processing, which is computationally prohibitive for real-time applications. Researchers at Microsoft have developed a novel approach that bypasses these limitations by decomposing complex 3D localization into a sequence of efficient 2D problems, achieving high accuracy and real-time performance in dynamic 4D point clouds.

Lightweight People Detection and Tracking

The first stage of the pipeline involves identifying individuals within the 3D volume. Rather than performing a brute-force search, the system projects the 3D data into three specialized 2D feature maps: a top-down map ($f_t$) for height, a side-sum map ($f_s$) for body bulk, and a bottom-up map ($f_b$) for lower-body features. These maps are processed by a modified AlexNet to quickly propose 3D bounding boxes for each person.

The system identifies and tracks multiple individuals in a dense environment using lightweight 2D projections.

To maintain identity across frames, the authors utilize a "max-covering" bipartite matching formulation. This ensures that trajectories remain stable even when the detector misses a frame or when individuals pass in close proximity to one another.

Hand Localization via Decomposition

The core innovation of this research is the decomposition of hand localization. Instead of a heavy 3D regressor, the system follows a three-step process:

  1. Horizontal Localization: A Fully Convolutional Network (FCN) analyzes the top-down view of a person's sub-volume to find $(x, y)$ coordinates.
  2. Slicing: The system extracts "thin volumes" (vertical slices) at those specific $(x, y)$ locations.
  3. Vertical Localization: Side-view FCNs process these slices to determine the $z$ (height) coordinate.
Hand localization results showing 3D bounding boxes and accurately localized hand positions represented by red and green spheres.

This decomposition allows the model to leverage body context (such as shoulder position) to "hallucinate" hand locations even when they are visually merged with other body parts, a significant advantage over geometric triangulation.

Key Results and Insights

The proposed method demonstrates that 2D projections retain sufficient spatial information to solve 3D problems efficiently. Key performance metrics include:

  • High Accuracy: The method achieved a mean error of 5.22 voxels (~10.4 cm), nearly halving the error of multi-view OpenPose approaches (11.11 voxels).
  • Real-time Speed: Localization takes only 5 milliseconds per person on a standard GPU.
  • Scalability: The architecture is efficient enough to track hundreds of people simultaneously across up to 50 cameras.

While the system is robust against sensor noise, it currently focuses on localization rather than full hand pose (finger orientation). Future work may extend the "thin volume" slice technique to enable detailed finger tracking within the same real-time framework.

Summary updated

Abstract

We propose novel real-time algorithm to localize hands and find their associations with multiple people in the cluttered 4D volumetric data (dynamic 3D volumes). Different from the traditional multiple view approaches, which find key points in 2D and then triangulate to recover the 3D locations, our method directly processes the dynamic 3D data that involve both clutter and crowd. The volumetric representation is more desirable than the partial observations from different view points and enables more robust and accurate results. However, due to the large amount of data in the volumetric representation brute force 3D schemes are slow. In this paper, we propose novel real-time methods to tackle the problem to achieve both higher accuracy and faster speed than previous approaches. Our method detects the 3D bounding box of each subject and localizes the hands of each person. We develop new 2D features for fast candidate proposals and optimize the trajectory linking using a new max-covering bipartite matching formulation, which is critical for robust performance. We propose a novel decomposition method to reduce the key point localization in each person 3D volume to a sequence of efficient 2D problems. Our experiments show that the proposed method is faster than different competing methods and it gives almost half the localization error.
BibTeX
@misc{jiang2019real,
  author = {Hao Jiang and Quanzeng You},
  title = {{Real-time Multiple People Hand Localization in 4D Point Clouds}},
  year = {2019},
  eprint = {1903.01695},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1903.01695},
}
Figure