Shape Tracking with Occlusions via Coarse-to-Fine Region-Based Sobolev Descent.

Yanchao Yang, Ganesh Sundaramoorthi

TL;DR

This paper introduces a novel method for tracking the precise shape of objects in videos, even when they are partially hidden (occluded) or reappear (dis-occluded). It achieves this by developing a sophisticated mathematical framework that models these occlusions and uses a "coarse-to-fine" optimization approach to ensure accurate tracking across frames.

What Problem Does This Paper Address?

Tracking objects in videos is crucial for many applications, from video editing to robotics. A common challenge is when parts of an object become hidden from view (occlusion) or reappear after being hidden (dis-occlusion) due to the object's movement or camera motion. Existing methods that rely on global image statistics or simple shape/appearance models struggle to adapt to these changes, leading to inaccurate tracking. This paper specifically tackles the problem of accurately tracking object shapes in the presence of these self-occlusions and dis-occlusions, which are particularly problematic for methods that jointly model shape and appearance.

Methods

The core of the paper lies in two main contributions:

  • Modeling Occlusions and Dis-occlusions: The authors formulate a joint optimization problem that simultaneously estimates the object's shape, its appearance (radiance), the warp (deformation) needed to track it, and crucially, the occluded and dis-occluded regions. They introduce a prior based on image similarity to detect dis-occluded regions.
  • Coarse-to-Fine Sobolev Descent: To efficiently solve this complex optimization problem, they introduce a novel "coarse-to-fine" optimization scheme. This is achieved by defining a new infinite-dimensional Riemannian manifold for parameterized regions and a novel Sobolev-type metric on this manifold. The gradient descent on this manifold naturally favors large-scale deformations first, then gradually refines them to finer details, making it robust and efficient for tracking. This approach is parameter-free, meaning it doesn't require manual tuning of regularization parameters.

Datasets

The paper evaluates its method on various video sequences exhibiting self-occlusions, dis-occlusions, complex object radiance, and cluttered backgrounds. Specific examples mentioned include sequences with a man walking, a fish, and a skater. The authors also compare their results to existing methods using their performance on these challenging videos. No new datasets were introduced.

Key Insights and Ideas

  • Occlusions are not just a nuisance, but a solvable problem: The paper demonstrates that explicitly modeling occlusions and dis-occlusions is critical for robust object tracking, rather than trying to ignore or work around them.
  • The power of a well-chosen mathematical framework: By defining a novel Riemannian manifold and a Sobolev metric, the authors achieve an automatic "coarse-to-fine" optimization that is inherently beneficial for tracking. This avoids the need for ad-hoc multi-scale strategies.
  • Joint optimization is key: Coupling the estimation of shape, appearance, warp, and occlusions leads to more accurate and robust results than treating these as separate problems.
  • Parameter-free optimization is desirable: The proposed Sobolev descent method's ability to avoid manual parameter tuning is a significant practical advantage.

Key Results

The paper demonstrates superior shape accuracy compared to recent methods, especially in challenging scenarios involving occlusions, dis-occlusions, complex object radiance, and cluttered backgrounds. Qualitative results show that their method successfully tracks objects where other methods fail due to these challenges. Quantitative analysis using F-measure shows their method consistently outperforms Scribbles and Adobe After Effects (AAE) on various sequences. For instance, on the "Lady Mercedes" sequence, Scribbles only tracks the head, while their method captures the entire object. The paper also shows that their method is robust to variations in key parameters like occlusion and dis-occlusion thresholds.

Limitations and Future Work

The authors acknowledge a limitation in their dis-occlusion detection: it may fail to detect dis-occluded parts of an object that do not share similar radiance with the current template (e.g., the sole of a shoe). Future work is suggested to include handling full occlusions of an object by other objects and further improving dis-occlusion detection.

Summary updated

Abstract

We present a method to track the shape of an object from video. The method uses a joint shape and appearance model of the object, which is propagated to match shape and radiance in subsequent frames, determining object shape. Self-occlusions and dis-occlusions of the object from camera and object motion pose difficulties to joint shape and appearance models in tracking. They are unable to adapt to new shape and appearance information, leading to inaccurate shape detection. In this work, we model self-occlusions and dis-occlusions in a joint shape and appearance tracking framework. Self-occlusions and the warp to propagate the model are coupled, thus we formulate a joint optimization problem. We derive a coarse-to-fine optimization method, advantageous in tracking, that initially perturbs the model by coarse perturbations before transitioning to finer-scale perturbations seamlessly. This coarse-to-fine behavior is automatically induced by gradient descent on a novel infinite-dimensional Riemannian manifold that we introduce. The manifold consists of planar parameterized regions, and the metric that we introduce is a novel Sobolev metric. Experiments on video exhibiting occlusions/dis-occlusions, complex radiance and background show that occlusion/dis-occlusion modeling leads to superior shape accuracy.
BibTeX
@article{yang2015shape,
  author = {Yanchao Yang and Ganesh Sundaramoorthi},
  title = {{Shape Tracking with Occlusions via Coarse-to-Fine Region-Based Sobolev Descent.}},
  journal = {IEEE Trans. Pattern Anal. Mach. Intell.},
  year = {2015},
  url = {https://openreview.net/forum?id=HmK4yjNeO6S},
}
Figure