TL;DR
This paper introduces a novel visual tracking method that tackles challenges like occlusion and appearance changes by treating the target as a collection of patches. It leverages "joint sparse representation" to model temporal consistency across patches and frames, and a new occlusion detection scheme to improve dictionary updates, leading to state-of-the-art performance on challenging datasets.
What Problem Does This Paper Address?
Visual tracking, a fundamental problem in computer vision, aims to follow an object's bounding box across video frames. However, real-world scenarios present significant challenges, primarily target appearance change (due to illumination, pose, etc.) and occlusion (when the target is hidden by other objects). Existing methods often struggle with these issues, leading to tracking failures or "drift" where the tracker loses the target. While sparse representation has shown promise, existing approaches often treat the target as a single entity or don't effectively handle occlusions, which can corrupt the learned appearance model. This paper aims to create a more robust tracker by specifically addressing these limitations through a patch-based approach and improved occlusion handling.
Methods
The proposed method, "Patchwise Joint Sparse Tracking with Occlusion Detection" (PJS-M/PJS-S), builds upon particle filter-based tracking. Here's a breakdown of its key components:
- Patchwise Representation: Instead of treating the entire target as one unit, the target is divided into several non-overlapping patches. This allows for more localized handling of appearance changes and occlusions.
- Appearance Model (Dictionary Learning): The appearance of each patch is modeled using a dictionary composed of corresponding patches from previous "best" candidate frames. This dictionary is learned using sparse representation.
- Joint Sparse Representation: A core innovation is the assumption that the current target patch and its corresponding patches from a few previous best candidates belong to a common subspace. This is enforced by solving a "joint sparse representation" problem, which encourages these patches to share a similar sparsity pattern. This helps maintain temporal consistency and robustness to appearance changes.
- Patchwise Likelihood Measure: The likelihood of a candidate being the true target is calculated as the sum of reconstruction errors across all its patches. This patch-based approach makes the tracker more resilient to partial occlusions, as long as some patches remain visible.
- Occlusion Detection: A novel occlusion detection scheme is introduced. It uses patch-reconstruction errors and an adaptive Markov chain to estimate the probability of a patch being occluded.
- Dictionary Update: Crucially, occluded patches identified by the detection scheme are excluded from the dictionary update process. This prevents corrupted information from degrading the appearance model and causing drift.
Datasets
The paper evaluates its method on a set of 10 publicly available datasets known to exhibit common tracking challenges: board, crossing, david, dollar, faceocc2, skating1, stone, sylv, trellis, and walking2. These datasets were annotated with attributes like illumination variation, scale variation, pose change, and partial/severe occlusion. No new datasets were introduced.
Key Insights and Ideas
- Patching is Powerful for Occlusion: Treating the target as a collection of patches is a key insight. It allows the tracker to isolate the impact of occlusion to specific parts of the target, rather than letting it corrupt the entire model.
- Temporal Consistency via Joint Sparsity: The idea of enforcing "joint sparsity" across patches and frames is a clever way to leverage temporal information. It assumes that the underlying structure of the target's appearance remains consistent over short periods, even with minor changes, and this shared structure can be learned.
- Occlusion Detection is Crucial for Dictionary Maintenance: Simply updating the dictionary with the best candidate can be disastrous if that candidate is occluded. The proposed occlusion detection and exclusion mechanism is vital for maintaining a clean and accurate appearance model over time.
- Likelihood based on Reconstruction Error: Using the sum of patch-reconstruction errors as a likelihood measure is intuitive. A good candidate should be well-represented by the learned appearance model, meaning low reconstruction error.
Key Results
The proposed method, PJS-M and PJS-S, demonstrated state-of-the-art performance across the evaluated datasets.
- Quantitative: PJS-S and PJS-M achieved the lowest Center Location Error (CLE) and highest PASCAL overlap score (VOC) on average. They consistently ranked among the top performers, often securing the first or second place. The average success rate for PJS-S and PJS-M was 69% and 65%, respectively.
- Qualitative: The paper presents qualitative results showing the proposed trackers successfully handling challenging scenarios where other state-of-the-art trackers failed, particularly in cases of severe occlusion, rapid pose changes, and illumination variations. For instance, in the "board" sequence, only PJS-S tracked the target throughout. In "david," it showed robustness to pose changes and illumination variations.
Limitations and Future Work
- Limitations:
- The paper notes that while the temporal similarity assumption improves performance against appearance change, it's less profound for video sequences with rapid changes.
- The simple case of equal variance for all patches was used in experiments, suggesting that more sophisticated variance settings (e.g., proportional to distance from the rectangle center) could further improve results.
- The joint sparse representation assumption might be violated with very fast movements or extreme pose changes, leading to performance degradation.
- Future Work:
- Improving tracker speed for real-time applications.
- Exploring weighted sums of patch-reconstruction errors to give less weight to marginal patches that are more prone to background and occlusion.
- Investigating the effect of different group sizes (k) in the joint sparse representation, as shown in Figure 11, where larger k generally improved performance but could be detrimental if the temporal similarity assumption was violated.
Summary updated