A Convolutional Neural Network Hand Tracker

Steven J. Nowlan, John C. PlattNeurIPS 1994

Introduction

Hand tracking and gesture recognition in natural, cluttered settings present significant computer vision challenges. Unlike handwritten digits, which possess a relatively consistent structure once aligned, human hands are highly articulated and can appear in a near-infinite variety of orientations.

To illustrate this challenge, consider that averaging thousands of images of a specific digit yields a blurry but still recognizable shape.

Averaged handwritten digits retain a recognizable structure, contrasting with the high variability of hand gestures.

However, averaging images of an open hand results in a featureless blob due to extreme variations in orientation and finger positioning. This high intra-class variability causes traditional template matching to fail. This paper explores whether Convolutional Neural Networks (CNNs)—renowned for their translation invariance—can overcome this lack of clean structure to track hands and recognize simple gestures in real-time, without relying on specialized hardware or controlled lighting.

System Architecture and Methodology

The proposed system is divided into two primary subsystems: Hand Tracking and Gesture Recognition, designed to operate efficiently on 1990s-era hardware by strictly limiting the architecture to convolutional layers and simple image filters.

Dual-Stream Localization

To locate the hand, the system processes sub-sampled video frames using two parallel CNNs:

  • Intensity Hand Locator Network: Analyzes the raw grayscale image to find hand-like structures.
  • Difference Hand Locator Network: Analyzes a "difference frame" (created by subtracting the previous frame from the current one). This acts as a crude velocity signal, leveraging the prior that hands are typically the most salient moving objects in a user-interface context.

Temporal Voting

Rather than relying on a single frame's prediction, the system employs a robust voting procedure. It combines the localization hypotheses from both the Intensity and Difference networks with a trajectory prediction calculated from the hand's position in the three preceding frames. This hybrid prediction method ensures temporal consistency and prevents the tracker from "jumping" to false positives in background clutter.

Gesture Classification

Once the hand's position is isolated, the system crops that specific region of the frame and feeds it into a dedicated Open/Close Network. This network evaluates the cropped region to classify the hand's state as either open or closed.

Experimental Setup and Results

The system was evaluated on a diverse test set designed to demonstrate user and environmental independence. The dataset featured 18 different individuals across 18 distinct room environments, capturing natural clutter and variable lighting conditions using black-and-white, $320 \times 240$ pixel digitized video frames.

The performance of the system was remarkably high:

  • Tracking Precision: The system successfully located the hand within $\pm 10$ pixels of its true position in 99.7% of the test frames.
  • Gesture Accuracy: The Open/Close Network correctly identified the hand's state in 99.1% of the frames.
  • Efficiency: The architecture successfully achieved real-time operation on contemporary hardware.

Limitations and Future Directions

While groundbreaking for its time, the system has several limitations. First, it only distinguishes between binary gestures ("open" and "closed"), serving primarily as a baseline to see if CNNs can handle complex classification where simple segmentation fails. Second, the architecture was heavily constrained by the hardware of the era; modern implementations could easily leverage deeper networks or more sophisticated optical flow techniques. Finally, the current model focuses exclusively on tracking a single hand and does not address multi-hand tracking or complex hand-over-hand occlusions.

Summary updated

BibTeX
@inproceedings{nowlan1994convolutional,
  author = {Steven J. Nowlan and John C. Platt},
  title = {{A Convolutional Neural Network Hand Tracker}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {1994},
}
Figure