Modeling Shape, Appearance and Self-Occlusions for Articulated Object Tracking

Yanchao Yang, Ganesh Sundaramoorthi

Abstract

We present a method to track the precise shape of an object in video based on new modeling and optimization on a new Riemannian manifold of parameterized regions. Joint dynamic shape and appearance models, in which a template of the object is propagated to match the object shape and radiance in the next frame, are advantageous over methods employing global image statistics in cases of complex object radiance and cluttered background. In cases of 3D object motion and viewpoint change, self-occlusions and dis-occlusions of the object are prominent, and current methods employing joint shape and appearance models are unable to adapt to new shape and appearance information, leading to inaccurate shape detection. In this work, we model self-occlusions and dis-occlusions in a joint shape and appearance tracking framework. Self-occlusions and the warp to propagate the template are coupled, thus a joint problem is formulated. We derive a coarse-to-fine optimization scheme, advantageous in object tracking, that initially perturbs the template by coarse perturbations before transitioning to finer-scale perturbations, traversing all scales, seamlessly and automatically. The scheme is a gradient descent on a novel infinite-dimensional Riemannian manifold that we introduce. The manifold consists of planar parameterized regions, and the metric that we introduce is a novel Sobolev-type metric defined on infinitesimal vector fields on regions. The metric has the property of resulting in a gradient descent that automatically favors coarse-scale deformations (when they reduce the energy) before moving to finer-scale deformations. Experiments on video exhibiting occlusion/dis-occlusion, complex radiance and background show that occlusion/dis-occlusion modeling leads to superior shape accuracy compared to recent methods employing joint shape/appearance models or employing global statistics.

Introduction

Precise shape tracking, often referred to as rotoscoping, is a fundamental task in film post-production and high-end video editing. Traditional methods frequently struggle with two primary challenges: global statistics failure and 3D motion complexities. Global statistics methods rely on simple foreground/background distributions, which often lead to "leakage" in cluttered scenes where objects share similar colors with their surroundings. Furthermore, when objects move in 3D, self-occlusions and dis-occlusions occur. Standard template-matching trackers cannot adapt to these appearance changes, causing the tracker to drift as it attempts to fit an outdated template to a new shape.

This research addresses the "chicken-and-egg" problem of registration and occlusion: an accurate warp is required to identify occlusions, yet knowledge of occlusions is necessary to compute an accurate warp. We propose a framework that jointly models shape, appearance, and self-occlusions on a novel Riemannian manifold.

Methodology

The proposed framework is built upon a joint shape and appearance tracking model that propagates a template to match the object's shape and radiance in subsequent frames.

Riemannian Manifold and Sobolev Metric

The core innovation is modeling the object shape as a point on an infinite-dimensional Riemannian manifold of parameterized regions. Unlike standard $L^2$ metrics that treat pixel perturbations independently, we introduce a Sobolev-type metric. This metric defines the cost of deformation based on smoothness, naturally inducing a gradient descent that prioritizes coarse-scale global motions (like translation) before refining local details. This "coarse-to-fine" behavior emerges automatically from the geometry of the space, eliminating the need for manual parameter tuning or image pyramids.

Joint Occlusion and Dis-occlusion Modeling

We formulate an energy function where the warp ($w$) and the occluded region ($O$) are optimized simultaneously. Parts of the template that fail to match the target frame are identified as occluded and excluded from the warping calculation.

Joint modeling of warp, occlusion, and dis-occlusion.

To handle dis-occlusions—parts of the object that were previously hidden—the model employs a self-similarity prior. It searches for regions near the current boundary that match the radiance statistics of the existing template. This effectively transforms the tracking of new parts into a localized detection problem.

Dis-occlusion detection and integration.

Experimental Results

The method was evaluated on several challenging video sequences featuring complex motion, such as "Library," "Hobbit," and "Lady." Comparisons were made against industry-standard tools like Adobe After Effects CS6 and state-of-the-art trackers using global statistics.

Quantitative Performance

In terms of F-measure, our method consistently outperformed competitors. For instance, in the "Lady" sequence—characterized by dark clothing against a dark background—our tracker achieved an F-measure of 0.95, whereas global statistics methods dropped to 0.29 due to background leakage.

Qualitative Robustness

The tracker successfully maintained precise boundaries during complex articulated motions, such as legs crossing or arms reappearing from behind a torso. The parameter-independent nature of the Sobolev optimization allowed the same settings to be effective across diverse video types, from swimming fish to human movement in cluttered environments.

Conclusion and Future Work

We have presented a principled framework for articulated object tracking that explicitly models self-occlusions and dis-occlusions. By leveraging a novel Riemannian manifold and Sobolev metric, we achieve a natural coarse-to-fine optimization that is robust to clutter and 3D motion.

While effective, the current implementation is not real-time, requiring approximately 8 seconds per HD frame. Future work will focus on computational optimizations, handling full occlusions by external objects, and improving dis-occlusion detection for parts with radically different appearances from the initial template.

Summary updated

Abstract

We present a method to track the precise shape of an object in video based on new modeling and optimization on a new Riemannian manifold of parameterized regions. Joint dynamic shape and appearance models, in which a template of the object is propagated to match the object shape and radiance in the next frame, are advantageous over methods employing global image statistics in cases of complex object radiance and cluttered background. In cases of 3D object motion and viewpoint change, self-occlusions and dis-occlusions of the object are prominent, and current methods employing joint shape and appearance models are unable to adapt to new shape and appearance information, leading to inaccurate shape detection. In this work, we model self-occlusions and dis-occlusions in a joint shape and appearance tracking framework. Self-occlusions and the warp to propagate the template are coupled, thus a joint problem is formulated. We derive a coarse-to-fine optimization scheme, advantageous in object tracking, that initially perturbs the template by coarse perturbations before transitioning to finer-scale perturbations, traversing all scales, seamlessly and automatically. The scheme is a gradient descent on a novel infinite-dimensional Riemannian manifold that we introduce. The manifold consists of planar parameterized regions, and the metric that we introduce is a novel Sobolev-type metric defined on infinitesimal vector fields on regions. The metric has the property of resulting in a gradient descent that automatically favors coarse-scale deformations (when they reduce the energy) before moving to finer-scale deformations. Experiments on video exhibiting occlusion/dis-occlusion, complex radiance and background show that occlusion/dis-occlusion modeling leads to superior shape accuracy compared to recent methods employing joint shape/appearance models or employing global statistics.
BibTeX
@article{yang2015modeling,
  author = {Yanchao Yang and Ganesh Sundaramoorthi},
  title = {{Modeling Shape, Appearance and Self-Occlusions for Articulated Object Tracking}},
  journal = {IEEE Trans. Pattern Anal. Mach. Intell.},
  year = {2015},
  url = {https://openreview.net/forum?id=HmK4yjNeO6S},
}

Loading related content...

Figure