Modeling Self-Occlusions in Dynamic Shape and Appearance Tracking

Yanchao Yang, Ganesh SundaramoorthiICCV 2013 · OpenReview

Abstract

We present a method to track the precise shape of a dynamic object in video. Joint dynamic shape and appearance models, in which a template of the object is propagated to match the object shape and radiance in the next frame, are advantageous over methods employing global image statistics in cases of complex object radiance and cluttered background. In cases of complex 3D object motion and relative viewpoint change, self-occlusions and dis-occlusions of the object are prominent, and current methods employing joint shape and appearance models are unable to accurately adapt to new shape and appearance information, leading to inaccurate shape detection. In this work, we model self-occlusions and dis-occlusions in a joint shape and appearance tracking framework. Experiments on video exhibiting occlusion/dis-occlusion, complex radiance and background show that occlusion/dis-occlusion modeling leads to superior shape accuracy compared to recent methods employing joint shape/appearance models or employing global statistics.

Introduction

Precise silhouette tracking in dynamic video is a fundamental challenge in computer vision. Traditional approaches often rely on global image statistics, such as color histograms, to differentiate foreground from background. While robust in some scenarios, these methods frequently fail when an object's appearance blends into a cluttered or similarly colored background.

An alternative is joint shape and appearance matching, which warps a template to match subsequent frames. However, these models are often "brittle" when faced with complex 3D motion. As an object rotates or moves, parts of it naturally disappear (self-occlusion) or new parts emerge (dis-occlusion). Standard warping algorithms struggle to adapt, leading to "drift" and tracking failure. This paper introduces a framework that explicitly models these visibility changes to maintain an accurate tracking template.

Figure 1: Framework overview showing occlusion ($O_t$), warping ($w_t$), and dis-occlusion ($D_{t+1}$) during a walking sequence.

Methods

The proposed framework evolves both the object's silhouette (shape) and its pixel values (radiance) through a multi-stage process:

  1. Non-Rigid Deformation: Large object deformations are modeled using a time-varying vector field (diffeomorphism), allowing the template to stretch and bend smoothly to match the target.
  2. Joint Occlusion Detection: Occlusion detection is coupled with deformation optimization. If a template region cannot be mapped to the next frame without significant radiance error, it is labeled as "occluded" and removed from the active template.
  3. Dis-occlusion via Self-Similarity: To identify newly visible parts (dis-occlusions), the authors employ a self-similarity prior. They assume new regions will resemble existing parts of the object. By analyzing local image similarity and proximity to the current boundary, the algorithm can "grow" the shape to incorporate these new areas.
  4. Recursive Filtering: The radiance model is updated over time using a recursive filter. This allows the tracker to adapt to illumination changes and non-Lambertian reflections without losing historical appearance data.
Figure 2: Qualitative result on the 'Hobbit' sequence showing the addition of dis-occluded regions to the shape model.

Experimental Evaluation

The method was evaluated on several challenging sequences, including "Lady Mercedes," "Marple," "Skater," and "Hobbit." These sequences feature high motion, non-rigid deformation, and significant viewpoint changes where legs cross or bodies turn.

Key Results

  • Quantitative Performance: The proposed method consistently outperformed competitors in F-measure scores. On the "Lady" sequence, it achieved a score of 0.9508, compared to 0.8243 for Adobe After Effects (AAE) CS6 and 0.2986 for the global-statistics-based "Scribbles" method.
  • Qualitative Robustness: The tracker maintained precise silhouettes even during complex self-occlusions (e.g., walking legs) where other methods typically "leak" into the background.
  • Efficiency: The C++ implementation processes 720p HD video at approximately 8 seconds per frame on a standard CPU, making it suitable for high-quality offline post-production.

Discussion and Future Work

The core insight of this work is that occlusion and motion are intrinsically linked; solving them jointly prevents the errors that plague independent estimation. By leveraging self-similarity, the model handles 3D rotations without requiring complex 3D CAD models.

However, limitations remain. The dis-occlusion detection depends on visual similarity; if a rotation reveals a drastically different texture (e.g., a hidden logo), the algorithm may struggle. Additionally, while the model handles self-occlusion, full occlusion by external objects remains a separate challenge. Future work may explore GPU acceleration to move toward interactive or real-time performance.

Summary updated

Abstract

We present a method to track the precise shape of a dynamic object in video. Joint dynamic shape and appearance models, in which a template of the object is propagated to match the object shape and radiance in the next frame, are advantageous over methods employing global image statistics in cases of complex object radiance and cluttered background. In cases of complex 3D object motion and relative viewpoint change, self-occlusions and dis-occlusions of the object are prominent, and current methods employing joint shape and appearance models are unable to accurately adapt to new shape and appearance information, leading to inaccurate shape detection. In this work, we model self-occlusions and dis-occlusions in a joint shape and appearance tracking framework. Experiments on video exhibiting occlusion/dis-occlusion, complex radiance and background show that occlusion/dis-occlusion modeling leads to superior shape accuracy compared to recent methods employing joint shape/appearance models or employing global statistics.
BibTeX
@inproceedings{yang2013modeling,
  author = {Yanchao Yang and Ganesh Sundaramoorthi},
  title = {{Modeling Self-Occlusions in Dynamic Shape and Appearance Tracking}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2013},
  url = {https://openreview.net/forum?id=r1b4fbG_ZS},
}
Figure