Model-Based Tracking of Self-Occluding Articulated Objects

James M. Rehg, Takeo KanadeICCV 1995 · OpenReview

Abstract

Computer sensing of hand and limb motion is an important problem for applications in human computer interaction and computer graphics. We describe a framework for local tracking of self-occluding motion, in which one part of an object obstructs the visibility of another. Our approach uses a kinematic model to predict occlusions and windowed templates to track partially occluded objects. We present offline 3D tracking results for hand motion with significant self-occlusion.

Introduction: The Visibility Problem

Tracking human hands and limbs is notoriously difficult because they are highly articulated and frequently "self-occlude." For instance, when a hand makes a fist, fingers disappear behind the palm; when fingers cross, one blocks the other. Standard tracking algorithms often fail in these scenarios because they cannot distinguish between a part moving away and a part simply being hidden. This core challenge is known as the visibility problem: the difficulty of assigning image pixels to the correct part of a 3D model when multiple parts overlap from the camera's perspective.

Visual representation of the three primary occlusion states: (a) first finger occluding second, (b) disjoint, and (c) second finger occluding first.

Methodology

The proposed framework utilizes a "top-down" approach, combining 3D kinematic geometry with 2D image processing to resolve ambiguities.

Kinematic Modeling and Layered Templates

The hand is represented as a system of rigid bodies (links) connected by joints with specific degrees of freedom (DOFs). Each rigid link is assigned a 2D template. These templates are treated as "layers," similar to a graphics program. To handle complex interactions, the system constructs a directed occlusion graph where an edge from Part A to Part B signifies that A occludes B. A topological sort on this graph establishes a definitive rendering order for the templates.

Occlusion Invariants and Predictive Visibility

A key innovation is the concept of regions in the "state space" (the set of possible hand configurations) where the visibility order of parts remains constant. By understanding the maximum possible movement between frames, the tracker can identify "safe" zones where the occlusion order is guaranteed to remain stable, enhancing robustness.

Geometric state space diagram showing how viewing direction determines the occlusion relationship between two fingers.

SSD Residual Minimization

Tracking is achieved by minimizing the Sum of Squared Differences (SSD) between a composite image—formed by the ordered templates—and the actual video frame. "Window functions" are employed to ensure that occluded pixels in background layers do not contribute to the error calculation, allowing the tracker to extract information solely from visible regions.

Experimental Results

The system was evaluated on a custom 80-frame sequence digitized from videotape. The sequence featured a hand where the index finger curled into the palm while the middle finger remained stationary. A 9-DOF kinematic model (representing two fingers with three planar joints each, plus global translation) was used.

The system successfully maintained 3D tracking despite significant self-occlusion. The "windowed template" approach proved robust, ignoring hidden parts of the template. Qualitative results showed the 3D cylinder model perfectly overlaid on video frames, even as the index finger disappeared behind the palm.

Discussion and Limitations

While foundational, the 1995 implementation faced several hurdles:

  • Computational Cost: The process was not real-time, requiring more advanced optimization (e.g., Gauss-Newton) and faster hardware.
  • Initialization: The system required pre-calibrated cameras and a known initial hand state.
  • Scaling: The experiment was limited to two fingers; scaling to a full 27-DOF hand model presents significant complexity in managing occlusion graphs and state space transitions.

Despite these limitations, the use of predictive visibility via 3D models remains a cornerstone of modern articulated object tracking.

Summary updated

Abstract

Computer sensing of hand and limb motion is an important problem for applications in human computer interaction and computer graphics. We describe a framework for local trading of self occluding motion, in which one part of an object obstructs the visibility of another. Our approach uses a kinematic model to predict occlusions and windowed templates to track partially occluded objects. We present offline 3D tracking results for hand motion with significant self occlusion.<
BibTeX
@inproceedings{rehg1995model,
  author = {James M. Rehg and Takeo Kanade},
  title = {{Model-Based Tracking of Self-Occluding Articulated Objects}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {1995},
  url = {https://openreview.net/forum?id=HkEyUZGubH},
}
Figure