Two-Stream Binocular Network: Accurate Near Field Finger Detection Based On Binocular Images

Yi Wei, Guijin Wang, Cairong Zhang, Hengkai Guo, Xinghao Chen, Huazhong Yang

TL;DR

The Two-Stream Binocular Network (TSBnet) bypasses the traditional, error-prone step of converting stereo images into depth maps for 3D fingertip tracking. By directly processing binocular images through a network featuring shared low-level layers and an embedded geometric measurement layer, TSBnet achieves an average 3D tracking error of just 10.9mm—outperforming previous direct-prediction methods by over 35%.

What Problem Does This Paper Address?

Accurate fingertip detection is a cornerstone of fluid human-computer interaction (HCI) and augmented reality (AR). However, tracking fingertips in 3D space is notoriously difficult due to their high range of motion and small surface area.

Traditionally, researchers solve this by taking binocular (stereo) camera images, calculating a dense depth map, and then applying depth-based hand pose estimation algorithms. This multi-stage pipeline has a glaring flaw: any noise or imprecision introduced during the depth map generation propagates directly into the fingertip coordinates. While some recent attempts have tried to predict 3D positions directly from stereo pairs, they struggle to generalize to complex, real-world hand movements.

Methods

The authors introduce TSBnet, a novel deep learning architecture designed to map raw binocular images directly to 3D fingertip and palm coordinates in an end-to-end fashion.

The complete pipeline of TSBnet, incorporating original and mask images through a two-stream architecture.

Key architectural innovations include:

  • Shared Low-Level Features: Because the left and right images of a stereo pair share basic visual structures (like edges and textures), TSBnet uses shared convolutional weights for early layers to save computation and improve feature consistency.
  • Two-Stream High-Level Extraction: The network splits into two distinct streams to extract high-level features unique to the left and right perspectives before fusing them in fully connected layers.
  • Multi-Channel Input: The network accepts four channels: the original left/right images (retaining rich detail) and left/right mask images (providing robustness against illumination changes and skin tones).
  • Differentiable Binocular Distance Measurement Layer: Instead of relying on post-processing, the authors embedded standard camera geometry equations directly into the network as a custom layer. This layer mathematically transforms predicted pixel coordinates and disparities into true 3D coordinates $(x, y, z)$.

The Training Trick

Directly training a network with an embedded geometric layer can cause gradients to explode during backpropagation due to the scaling factors involved in computing depth ($\frac{\lambda z}{f}$). To counter this, the authors developed a two-stage training strategy: they first pre-trained the network to predict 2D pixel coordinates and disparities, and then fine-tuned the entire model end-to-end using the 3D coordinate loss.

The two-stage training strategy designed to prevent exploding gradients.

Datasets

Publicly available hand pose datasets (like NYU, ICVL, and MSRA) are almost exclusively depth-based. Existing binocular datasets are small and typically feature only a single subject, making it impossible to test how well a model generalizes.

To validate TSBnet, the authors captured a substantial new binocular hand image dataset using a Leap Motion controller for images and a Trakstar magnetic tracker to establish highly accurate (1.4mm precision) 3D ground truth coordinates.

  • Training Set: ~117k image pairs from 8 individuals, covering 16 basic hand poses alongside extra variations.
  • Test Set: ~10k image pairs from 2 entirely different individuals, specifically featuring large ranges of rotation and motion to rigorously evaluate generalization.

Key Insights and Ideas

  • End-to-End Geometry Integration: The "aha moment" of the paper is recognizing that geometric camera principles don't need to be relegated to post-processing. By formalizing the math into a differentiable layer, the network can optimize its internal features specifically for the final 3D task.
  • Pre-training as a Regularizer for Exploding Gradients: The discovery that the geometric layer causes optimization instability led to a practical engineering insight: bounding the network's initial predictions to the 2D pixel space stabilizes the early learning phase, making subsequent 3D fine-tuning highly effective.

Key Results

TSBnet achieved an average 3D distance error of 10.9mm across all joints on the challenging test set. This significantly outperforms the previous state-of-the-art direct method (Chen et al.), which yielded a 16.8mm error—representing a 35.1% relative error reduction.

Performance comparison showing mean error distance per joint and the fraction of successful frames across distance thresholds.

An ablation study highlights how each incremental modification improved the baseline network:

  • Baseline (Original images only): 13.6 mm error
  • + Multi-scale training: 13.2 mm
  • + More channels: 12.3 mm
  • + Mask images: 11.4 mm
  • + Residual connections: 11.2 mm
  • + Binocular distance measurement layer: 10.9 mm

Qualitatively, the model proves exceptionally robust at tracking bent fingers (e.g., simulating a mouse click), a scenario where commercial alternatives like the default Leap Motion software frequently fail.

Limitations and Future Work

Not discussed in the paper.

Summary updated

Abstract

Fingertip detection plays an important role in human computer interaction. Previous works transform binocular images into depth images. Then depth-based hand pose estimation methods are used to predict 3D positions of fingertips. Different from previous works, we propose a new framework, named Two-Stream Binocular Network (TSBnet) to detect fingertips from binocular images directly. TSBnet first shares convolutional layers for low level features of right and left images. Then it extracts high level features in two-stream convolutional networks separately. Further, we add a new layer: binocular distance measurement layer to improve performance of our model. To verify our scheme, we build a binocular hand image dataset, containing about 117k pairs of images in training set and 10k pairs of images in test set. Our methods achieve an average error of 10.9mm on our test set, outperforming previous work by 5.9mm (relatively 35.1%).
BibTeX
@inproceedings{wei2017two,
  author = {Yi Wei and Guijin Wang and Cairong Zhang and Hengkai Guo and Xinghao Chen and Huazhong Yang},
  title = {{Two-Stream Binocular Network: Accurate Near Field Finger Detection Based On Binocular Images}},
  booktitle = {VCIP},
  year = {2017},
  doi = {10.1109/vcip.2017.8305146},
}
Figure