A Framework for Modeling Appearance Change in Image Sequences

Michael J. Black, David J. Fleet, Yaser YacoobICCV 1998 · OpenReview

Abstract

Image "appearance" may change over time due to a variety of causes such as: 1) object or camera motion; 2) generic photometric events including variations in illumination (e.g. shadows) and specular reflections; and 3) "iconic changes" which are specific to the objects being viewed and include complex occlusion events and changes in the material properties of the objects. We propose a general framework for representing and recovering these "appearance changes" in an image sequence as a "mixture" of different causes. The approach generalizes previous work on optical flow to provide a richer description of image events and more reliable estimates of image motion.

Introduction: Beyond Brightness Constancy

Traditional motion estimation in computer vision, often referred to as optical flow, has long relied on the brightness constancy assumption. This principle posits that the intensity of a pixel remains invariant as it moves from one frame to the next. However, real-world environments are dynamic and complex; shadows sweep across surfaces, metallic objects produce specular glints, and non-rigid objects like the human face constantly reveal new structures.

When these events occur, traditional algorithms often fail or treat the resulting intensity changes as noise. This paper argues that these changes are not nuisances but meaningful signals. By explicitly modeling these causes, we can achieve more robust motion tracking and a deeper understanding of the visual scene.

A Taxonomy of Appearance Change

The authors categorize appearance changes into four distinct physical and pictorial causes, as illustrated below:

Taxonomy of appearance changes: form, illumination, iconic, and specular.
  1. Form Change (Motion): Geometric transformations of the object, typically modeled as a warp using affine flow basis fields.
  2. Illumination Variations: Photometric shifts caused by global lighting changes or moving shadows, modeled as a smooth function scaling or offsetting image contrast.
  3. Specular Reflections: Near-saturation of image intensity common on reflective surfaces like metal or glass.
  4. Iconic Changes: Complex, object-specific events (e.g., teeth appearing during a smile) that are difficult to model with simple physics. These are captured statistically using Principal Component Analysis (PCA) to learn a subspace of possible appearances.

Methodology: The Probabilistic Mixture Model

To handle these diverse causes, the framework employs a probabilistic mixture model. Every pixel's change is explained by one or more of the generative models mentioned above. The system uses the Expectation-Maximization (EM) algorithm to iteratively solve for the parameters:

  • E-step: Calculate "ownership weights"—the posterior probability that a specific pixel's change was caused by a particular model (e.g., motion vs. illumination).
  • M-step: Update the parameters of each model (e.g., the affine warp parameters or PCA coefficients) to better fit the data based on these weights.

A robust likelihood function is integrated to ensure that extreme outliers do not destabilize the estimation process. This "soft assignment" approach allows the framework to handle regions where multiple causes overlap, such as a moving object that is also entering a shadow.

Experimental Results and Insights

The framework was validated across several challenging scenarios:

  • Shadow and Specularity Factorization: In sequences involving hands casting shadows or moving reflective staplers, the system successfully decoupled motion from photometric events. It could track the object's true motion while "factoring out" the shadow or identifying non-constant specular highlights.
  • Facial Expression Modeling: Using a training set of approximately 500 images of facial expressions, the authors demonstrated that iconic changes in the mouth region could be compactly represented. The first 11 PCA components accounted for 85% of the appearance variance, allowing the system to reconstruct teeth and tongue movements that pure motion warping cannot explain.
  • From Robustness to Explanation: A key insight of this work is the shift from making motion estimation "robust" to outliers toward explicitly "explaining" those outliers. This provides a much richer description of the image sequence.

Conclusion and Future Directions

This framework provides a powerful bridge between geometric motion estimation and data-driven appearance modeling. While the current implementation often focuses on mixtures of two causes at a time, future work aims to integrate all four models into a unified formulation. Additionally, incorporating spatial priors, such as Markov Random Fields, could improve the coherence of the explained regions, moving beyond the current pixel-independent assumption.

Summary updated

Abstract

Image "appearance" may change over time due to a variety of causes such as: 1) object or camera motion; 2) generic photometric events including variations in illumination (e.g. shadows) and specular reflections; and 3) "iconic changes" which are specific to the objects being viewed and include complex occlusion events and changes in the material properties of the objects. We propose a general framework for representing and recovering these "appearance changes" in an image sequence as a "mixture" of different causes. The approach generalizes previous work on optical flow to provide a richer description of image events and more reliable estimates of image motion.
BibTeX
@inproceedings{black1998framework,
  author = {Michael J. Black and David J. Fleet and Yaser Yacoob},
  title = {{A Framework for Modeling Appearance Change in Image Sequences}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {1998},
  url = {https://openreview.net/forum?id=ByNXjWMubB},
}
Figure