Shape Tracking With Occlusions via Coarse-To-Fine Region-Based Sobolev Descent

Yanchao Yang, Ganesh Sundaramoorthi

Abstract

We present a method to track the precise shape of an object in video based on new modeling and optimization on a new Riemannian manifold of parameterized regions. Joint dynamic shape and appearance models, in which a template of the object is propagated to match the object shape and radiance in the next frame, are advantageous over methods employing global image statistics in cases of complex object radiance and cluttered background. In cases of 3D object motion and viewpoint change, self-occlusions and dis-occlusions of the object are prominent, and current methods employing joint shape and appearance models are unable to adapt to new shape and appearance information, leading to inaccurate shape detection. In this work, we model self-occlusions and dis-occlusions in a joint shape and appearance tracking framework. Self-occlusions and the warp to propagate the template are coupled, thus a joint problem is formulated. We derive a coarse-to-fine optimization scheme, advantageous in object tracking, that initially perturbs the template by coarse perturbations before transitioning to finer-scale perturbations, traversing all scales, seamlessly and automatically. The scheme is a gradient descent on a novel infinite-dimensional Riemannian manifold that we introduce. The manifold consists of planar parameterized regions, and the metric that we introduce is a novel Sobolev-type metric defined on infinitesimal vector fields on regions. The metric has the property of resulting in a gradient descent that automatically favors coarse-scale deformations (when they reduce the energy) before moving to finer-scale deformations. Experiments on video exhibiting occlusion/dis-occlusion, complex radiance and background show that occlusion/dis-occlusion modeling leads to superior shape accuracy compared to recent methods employing joint shape/appearance models or employing global statistics.

Introduction: The Challenge of 3D Shape Tracking

Precise shape tracking is a fundamental task in computer vision, yet it remains exceptionally difficult when objects move in 3D space. As a person turns or a camera shifts its viewpoint, parts of the object inevitably disappear (self-occlusion) and new parts become visible (dis-occlusion).

Existing trackers typically fall into two categories, both with significant drawbacks. Global statistics-based trackers attempt to separate foreground from background using color or texture, but they often fail in cluttered environments or when the object shares visual characteristics with the background. Conversely, template-matching trackers warp an initial object model to subsequent frames. However, if the template does not account for topological changes caused by occlusion, the warp becomes increasingly inaccurate, leading to tracker drift.

Figure 1: Illustration of the tracking model. The template at time $t$ is propagated to time $t+1$ by accounting for occluded regions ($O_t$) and newly discovered dis-occluded regions ($D_{t+1}$).

Methodology: Joint Modeling and Sobolev Descent

The authors propose a joint optimization framework that simultaneously handles non-rigid warping, occlusion detection, and dis-occlusion estimation.

1. The Dynamic Model

The object at time $t+1$ is represented as a warped version of the previously visible parts plus a newly discovered region. This warp is modeled as a diffeomorphism on a novel Riemannian manifold of parameterized regions.

2. Sobolev Descent: The "Secret Sauce"

Standard optimization techniques, such as $L^2$ gradient descent, are inherently local and prone to noise or local minima. This paper introduces a Sobolev-type metric that penalizes high-frequency, "wiggly" changes in the deformation field. In practice, this metric forces the gradient descent to automatically favor coarse-scale deformations (like translation and rotation) before moving to fine-scale refinements (like finger movements). This "coarse-to-fine" behavior is a mathematical property of the Sobolev gradient, removing the need for manual parameter tuning or image pyramids.

3. The Tracking Pipeline

The system iterates through a multi-step process:

  • Warp/Occlusion Step: Finding the optimal smooth transformation while identifying regions that no longer match the template as "occluded."
  • Dis-occlusion Step: Using a self-similarity prior to grow the shape into newly visible areas that match the object's existing appearance.
Figure 3: The tracking pipeline illustrated on the 'Hobbit' sequence. (a) Input template and target. (b) Warped template and detected occlusion map. (c) Estimation of dis-occluded regions. (d) Final updated shape and appearance.

Experimental Results

The framework was evaluated on challenging video sequences featuring complex radiance and cluttered backgrounds. Quantitatively, the method consistently outperformed state-of-the-art global statistics trackers (like Scribbles) and commercial tools (Adobe After Effects CS6). For instance, on the "Lady" sequence, the proposed method achieved an F-measure of 0.9508, whereas Scribbles failed (0.2986) due to background similarities.

Qualitatively, the tracker demonstrated remarkable robustness. In the "Lady Mercedes" sequence, where a subject's pants match the appearance of car tires in the background, the Sobolev descent maintained the correct leg shape while other trackers "leaked" into the background.

Conclusion and Future Work

By explicitly modeling occlusions and leveraging the inherent multi-scale properties of Sobolev gradients, this work provides a robust solution for high-fidelity shape tracking. While the current implementation is computationally intensive (approx. 8 seconds per frame for 720p video), it offers superior accuracy in complex scenarios. Future research will focus on optimizing performance and extending the model to handle full occlusions by external objects.

Summary updated

Abstract

We present a method to track the precise shape of an object in video based on new modeling and optimization on a new Riemannian manifold of parameterized regions. Joint dynamic shape and appearance models, in which a template of the object is propagated to match the object shape and radiance in the next frame, are advantageous over methods employing global image statistics in cases of complex object radiance and cluttered background. In cases of 3D object motion and viewpoint change, self-occlusions and dis-occlusions of the object are prominent, and current methods employing joint shape and appearance models are unable to adapt to new shape and appearance information, leading to inaccurate shape detection. In this work, we model self-occlusions and dis-occlusions in a joint shape and appearance tracking framework. Self-occlusions and the warp to propagate the template are coupled, thus a joint problem is formulated. We derive a coarse-to-fine optimization scheme, advantageous in object tracking, that initially perturbs the template by coarse perturbations before transitioning to finer-scale perturbations, traversing all scales, seamlessly and automatically. The scheme is a gradient descent on a novel infinite-dimensional Riemannian manifold that we introduce. The manifold consists of planar parameterized regions, and the metric that we introduce is a novel Sobolev-type metric defined on infinitesimal vector fields on regions. The metric has the property of resulting in a gradient descent that automatically favors coarse-scale deformations (when they reduce the energy) before moving to finer-scale deformations. Experiments on video exhibiting occlusion/dis-occlusion, complex radiance and background show that occlusion/dis-occlusion modeling leads to superior shape accuracy compared to recent methods employing joint shape/appearance models or employing global statistics.
BibTeX
@article{yang2015shape,
  author = {Yanchao Yang and Ganesh Sundaramoorthi},
  title = {{Shape Tracking With Occlusions via Coarse-To-Fine Region-Based Sobolev Descent}},
  journal = {IEEE Trans. Pattern Anal. Mach. Intell.},
  year = {2015},
  url = {https://openreview.net/forum?id=HmK4yjNeO6S},
}
Figure