Abstract
We propose a Bayesian framework for representing and recognizing local image motion in terms of two primitive models: translation and motion discontinuity. Motion discontinuities are represented using a nonlinear generative model that explicitly encodes the orientation of the boundary, the velocities on either side, the motion of the occluding edge over time, and the appearance/disappearance of pixels at the boundary. We represent the posterior distribution over the model parameters given the image data using discrete samples. This distribution is propagated over time using the Condensation algorithm. To efficiently represent such a high-dimensional space we initialize samples using the responses of a low-level motion discontinuity detector.
Introduction
In classical computer vision, estimating image motion (optical flow) usually relies on a "smoothness assumption"—the idea that neighboring pixels move similarly. However, this assumption fundamentally breaks down at object boundaries, where one surface slides over another. Historically, researchers treated these boundaries as outliers or noise, leading to inaccurate flow estimation and the loss of vital cues about a scene's 3D structure, such as depth ordering (foreground vs. background).
This paper argues that motion discontinuities should be treated as explicit signals. By modeling the geometry and dynamics of these boundaries, we can better resolve depth ordering and accurately predict pixel appearance/disappearance between frames.
Generative Modeling of Motion
The proposed framework defines two primitive models for local image patches:
- Translation: A simple 2-parameter model for uniform motion.
- Motion Discontinuity: A complex 6-parameter generative model that encodes edge orientation, the velocities of both foreground and background surfaces, the edge's distance from the patch center, and the foreground/background assignment.
As shown in the figure above, the model explicitly accounts for occlusion and disocclusion. By modeling which pixels appear or disappear at a boundary, the likelihood calculation can ignore these regions, leading to more robust matching than standard flow algorithms.
Bayesian Tracking Framework
Because the 6-parameter space is high-dimensional and the probability distributions are often multi-modal (e.g., initial ambiguity about which side is the foreground), the system employs the Condensation algorithm (particle filtering). This represents the "belief" about motion as a set of discrete samples that evolve over time.
To prevent the particle filter from getting lost in high-dimensional space, the framework uses a "bottom-up" low-level motion edge detector to seed new particles in promising areas. This hybrid approach ensures the computational budget is spent on the most likely parts of the state space while maintaining multiple hypotheses until temporal data can disambiguate them.
Key Results and Discussion
The method was evaluated on classic sequences like the Pepsi can and the Flower Garden benchmark. Key findings include:
- Depth Ordering: The model successfully identifies foreground vs. background. In the Flower Garden sequence, it correctly assigns the tree to the foreground after a few frames of observation as the consistent interpretation dominates the posterior.
- Multi-modality: The sampling approach successfully maintains multiple "peaks" in the posterior distribution until more data arrives to resolve ambiguity.
- Model Switching: The system automatically switches from a discontinuity model to a translation model once an edge passes through a local neighborhood, demonstrating temporal flexibility.
Conclusion and Future Work
While effective, the current approach faces challenges in scaling. Modeling 6 parameters per local patch is computationally expensive, and the current model treats each neighborhood independently. Future work could involve a "spatial grammar" where neighboring patches inform each other's beliefs. Additionally, expanding the generative vocabulary to include rotation, scaling, and curved boundaries would improve performance in complex real-world scenes.
Summary updated