Abstract
Detailed tracking is required for many vision applications. A visual feature-based constraint underlies most conventional motion estimation methods. For example, optical flow methods assume that the brightness of each pixel is constant in two consecutive frames. However, it is difficult to realize accurate extraction and tracking using only visual feature information, because viewpoint changes and inconsistent illumination cause the visual features of some regions of objects to appear different in consecutive frames. A structure-based constraint of objects is also necessary for tracking. In the proposed method, both visual feature matching and structure matching are formulated as a linear assignment problem and then integrated.
Introduction
Most conventional motion estimation methods, such as optical flow, rely on the "brightness constancy" assumption—the idea that a pixel's brightness remains stable between consecutive frames. In real-world scenarios, this assumption is frequently violated by viewpoint changes, moving shadows, and specular reflections. Furthermore, objects with uniform color regions present the "aperture problem," where visually identical blocks make it impossible for standard algorithms to determine which part moved where.
To address these challenges, the authors propose a method that mimics human vision by considering not just color, but the stable spatial relationships (structure) between an object's parts. This approach ensures that tracking remains robust even when visual features are ambiguous or changing.
Methodology: Integrating Visual and Structural Constraints
The proposed approach enhances Exclusive Block Matching (EBM) with a structural component, formulated as a linear assignment problem.
- Visual Feature Matching: The image is divided into blocks, which are compared using HSV histograms and the Bhattacharyya distance to determine visual similarity.
- Structure Matching: This novel addition assumes that the relative positions of blocks within an object should not change abruptly. A "transformation cost" is calculated based on the square of the Euclidean distance between a block's current position and its expected position relative to its neighbors.
- Linear Assignment Formulation: Both visual and structural matching are treated as optimization problems. The system uses the Hungarian method (specifically the Munkres algorithm) to find the optimal set of matches that minimizes the total cost.
- Iterative Integration: Since simultaneous optimization is computationally expensive, the method uses an iterative approach. Initial visual matches estimate the object's structure, which then informs and "rewards" visual matches that align with that structure in subsequent loops.
Experimental Results
The proposed method (PM) was evaluated against standard techniques including Lucas-Kanade and SIFT-based tracking using both synthetic and real-world data.
- Error Reduction: In Computer Graphics (CG) tests involving rotation and magnification, the Relative Direction Error Rate (RDER)—a measure of internal structural warping—dropped from 6.34% in the original EBM to 3.20% with the new method.
- Robustness to Glare: Qualitative results showed that while Lucas-Kanade and SIFT failed when reflections hit the surface of a moving object, the proposed method maintained nearly perfect dense block flows.
- Aperture Problem Resolution: On real-world PETS 2006 benchmark data, the method successfully tracked individual parts of visually identical clothing by "tethering" them to neighboring blocks through structural constraints.
Limitations and Future Work
The authors identify two primary areas for improvement:
- Non-Rigid Motion: The current structure constraint assumes semi-rigid movement. Independent motion of parts (e.g., hands moving differently from the body) can introduce errors, suggesting a need for motion segmentation.
- Shadow Interference: Shadows moving in sync with objects are sometimes incorrectly extracted. Future work will focus on integrating specific shadow-detection algorithms to improve extraction accuracy.
Summary updated