Serial-parallel Multi-Scale Feature Fusion for Anatomy-Oriented Hand Joint Detection

Bin Li, Hong Fu, Ruimin Li, Wendi WangarXiv 2021 · arXiv

Abstract

This paper presents a robust two-stage framework for hand joint detection in unconstrained images containing multiple hands at various scales. By integrating serial and parallel multi-scale feature fusion (MSFF) with an anatomy-oriented reinforcement mechanism, the proposed model addresses common pitfalls such as scale variability and complex backgrounds. Achieving state-of-the-art accuracy (0.97 [email protected] on the DCD8-6000 dataset) and real-time performance (up to 37 FPS), the system demonstrates significant improvements over existing methods like CPM and HRnet.

Introduction

Hand joint detection is a cornerstone of human-computer interaction (HCI) and gesture recognition. However, "in-the-wild" scenarios present significant hurdles, including varying hand scales, multiple hands per image, and occlusions. Traditional models often rely on pre-cropped single-hand inputs, failing when faced with complex, unconstrained backgrounds where distinguishing joints from noisy environments is difficult.

Example of an unconstrained input image from the NZSL dataset.

The authors propose a system that moves beyond these limitations by employing a two-stage approach that first localizes hands and then performs precise joint detection using anatomical priors and multi-scale feature fusion.

Proposed Methodology

The architecture is divided into two primary stages designed to progressively refine hand feature representations.

The proposed two-stage architecture: (a) Overall structure, (b) MSFF module, (c) Convolution Block, and (d) Anatomy-oriented mutual reinforcement.

Stage I: Hand Localization and SSHFR

Initially, a pre-trained network identifies coarse hand regions. These regions are cropped and processed by the Shallow Spatial Hand Features Representation (SSHFR) module. This ten-layer CNN extracts 256-channel feature maps, ensuring the network focuses on essential spatial data before deep feature extraction begins.

Stage II: Serial-Parallel Multi-Scale Feature Fusion (MSFF)

The core innovation lies in cascaded MSFF modules. Each module processes features through three parallel branches at different resolutions to capture both global structure and local details.

  • Fusion & Transpose Layer: This layer uses a unique "concatenate-reshape-transpose" operation to link multi-scale features, ensuring local joint details are contextualized within the global hand structure.
  • Anatomy-Oriented Mutual Reinforcement: By modeling hand joints as a graph and utilizing an anatomical adjacency matrix, the system reinforces joint predictions. For instance, a high-confidence wrist detection helps refine the predicted positions of knuckles and fingertips.
  • Adaptive Loss Mechanism: The model adaptively weights the contribution of each MSFF module during training, accelerating convergence by prioritizing modules providing the most useful gradients.

Experimental Evaluation

The model was evaluated on five diverse datasets, including synthetic (RHD, HS) and natural (MPII, NZSL) imagery. Additionally, the authors introduced the DCD8-6000 dataset, containing 6,000 natural images of subjects performing fine motor tasks, providing a challenging real-world benchmark.

Key Results

The proposed model consistently outperformed baselines such as CPM, OpenPose-hand (OPH), and HRnet. On the DCD8-6000 dataset, it achieved a [email protected] of 0.97, compared to 0.92 for HRnet and 0.49 for CPM. Furthermore, the system maintains a high processing speed of 24 to 37 FPS on an NVIDIA RTX 2080 GPU, making it suitable for real-time HCI applications.

Discussion and Insights

A significant contribution is the formal integration of anatomical constraints as a prior. This graph-based reinforcement mimics human intuition in predicting occluded joints based on visible landmarks. Additionally, the authors identified a "spread" metric—the area of non-zero pixels in a predicted heatmap—which serves as a reliable indicator of detection confidence, offering a useful diagnostic for real-time systems.

Conclusion and Future Work

While highly effective, the model's performance is currently dependent on the initial localization in Stage I. Future research will focus on unifying localization and detection into a single-pass architecture to further reduce latency and exploring 3D pose estimation extensions. Testing the "spread" metric as a trigger for active learning also remains a promising avenue for improving model robustness.

Summary updated

Abstract

Accurate hand joints detection from images is a fundamental topic which is essential for many applications in computer vision and human computer interaction. This paper presents a two stage network for hand joints detection from single unmarked image by using serial-parallel multi-scale feature fusion. In stage I, the hand regions are located by a pre-trained network, and the features of each detected hand region are extracted by a shallow spatial hand features representation module. The extracted hand features are then fed into stage II, which consists of serially connected feature extraction modules with similar structures, called "multi-scale feature fusion" (MSFF). A MSFF contains parallel multi-scale feature extraction branches, which generate initial hand joint heatmaps. The initial heatmaps are then mutually reinforced by the anatomic relationship between hand joints. The experimental results on five hand joints datasets show that the proposed network overperforms the state-of-the-art methods.
BibTeX
@misc{li2021serial,
  author = {Bin Li and Hong Fu and Ruimin Li and Wendi Wang},
  title = {{Serial-parallel Multi-Scale Feature Fusion for Anatomy-Oriented Hand Joint Detection}},
  year = {2021},
  eprint = {2102.09757},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2102.09757},
}
Figure