When a visual tracking system follows an object across successive video frames, it relies on an internal model of what that object looks like. In traditional tracking frameworks, this model is holistic: the target inside a bounding box is treated as a single, indivisible block of pixels or a unified global feature vector.
While holistic models work well under ideal conditions, they possess a fundamental vulnerability: they fail catastrophically under partial occlusion. If an obstacle hides even a small portion of an object, a holistic tracker does not just experience a minor drop in confidence. Instead, the global visual signature changes so drastically that the tracker can no longer recognize the target, causing it to drift onto the background or lock onto the occluding obstacle itself.
To overcome this vulnerability, a prominent research direction decomposes holistic representations into localized, structurally constrained part models. By shifting from a single global template to a collaborative network of local features, trackers can isolate occluded regions and maintain a stable trajectory using only the visible fragments of a target.
The Flaw of Holistic Representations
To understand why holistic tracking fails during partial occlusion, consider how a standard template-matching algorithm computes similarity. As Amnon Shashua et al. demonstrate in Algebraic Set Kernels with Application to Inference Over Local Image Representations, treating an entire image patch as a single global vector means that any localized change—such as an obstacle covering an edge—corrupts the entire representation.
Holistic Tracking (Brittle):
[ Target Template ] <---> [ Partially Occluded Target ] = Low Similarity Match (Drift)
Part-Based Tracking (Robust):
[ Part 1 (Visible) ] <---> [ Part 1 (Visible) ] = High Match
[ Part 2 (Occluded)] <---> [ Obstacle Feature ] = Ignored / Low Weight
[ Part 3 (Visible) ] <---> [ Part 3 (Visible) ] = High Match
When an object becomes partially obscured, a holistic tracker attempts to match the combined appearance of the remaining target and the occluding obstacle against its original template. As observed by Ting Liu 0009 et al. in Real-time part-based visual tracking via adaptive correlation filters, this lack of spatial granularity forces the tracking filter to incorporate background noise. If the tracker updates its template using this corrupted data, it suffers from "template update pollution," permanently losing focus on the actual target.
Decomposing the Target into Local Parts
The core intuition of part-based tracking is to distribute the tracking responsibility across a collection of localized sub-regions. If an object is represented as a set of independent or semi-independent parts, an occlusion that hides 30% of the object will only corrupt 30% of the model. The remaining 70% of the parts can still match the target successfully.
This strategy can be implemented in several ways:
- Anchor-Based Decompositions: In crowded environments where targets frequently overlap, trackers can divide an object into explicit semantic components. For example, Hatice Gunes et al. in Tracking People in Crowds by a Part Matching Approach break the human body into five distinct segments. If the legs or torso are hidden by a barrier, the system relies on unoccluded regions like the head as a reliable "anchor" to preserve the target's identity.
- Salient Feature Voting: Rather than enforcing rigid geometric segments, trackers can identify distinct, high-contrast local descriptors across the target. Kwang Moo Yi et al. leverage this in Initialization-Insensitive Visual Tracking through Voting with Salient Local Features, utilizing a Generalized Hough Transform (GHT) voting scheme. Each visible, salient local feature votes for the object's center. When partial occlusion hides a subset of features, the remaining visible features still cast accurate votes, allowing the system to pinpoint the target center despite the missing data.
Enforcing Structural Constraints
While breaking an object into independent parts solves the occlusion problem, it introduces a new challenge: spatial disintegration. If local parts are tracked completely independently, they lose their global context. A part filter tracking a patch of a target's jacket might easily drift onto a similar-looking patch of background clutter.
As Tianzhu Zhang et al. explain in Partial Occlusion Handling for Visual Tracking via Robust Part Matching, independent part matching fails because it ignores the geometric structure of the target. To prevent individual parts from jumping to spatially impossible locations, a robust tracker must enforce a structural check or a part-to-whole constraint.
An early realization of this principle is found in the work of Nicholas D. H. Dowson et al. in Simultaneous Modeling and Tracking (SMAT) of Feature Sets. Their framework maintains a dynamic library of appearance exemplars alongside a global structural model. The structural model constantly cross-checks the relative positions of individual features, filtering out any feature updates that violate the expected geometric layout of the object.
[Part 1] ---- (Constrained Distance) ---- [Part 2]
\ /
\ /
(Constrained) (Constrained)
\ /
\ /
+-----------> [Part 3] <--------+
To formalize these spatial relationships, researchers often model the target as a topological graph. In Structure Preserving Object Tracking, Lu Zhang 0036 et al. treat object parts as nodes in a connected graph, optimizing their configurations via an online structured SVM. By penalizing configurations where the distance between parts deviates from the baseline object structure, the tracker ensures that even if a part's visual appearance matches a background element, it cannot drift there unless the entire graph moves with it.
Modern Realizations: Attention and Part-to-Whole Reasoning
Modern tracking architectures have evolved from explicit bounding-box partitions to implicit, pixel-level part modeling using deep networks, graphs, and attention mechanisms. These approaches preserve the core intuition of part-based tracking—isolating occlusions while maintaining structural integrity—but execute it dynamically within neural feature spaces.
For instance, Dongyan Guo et al. introduce a Graph Attention Module in Graph Attention Tracking. Instead of performing a coarse cross-correlation between a template feature map and a search region, their tracker models the interaction as a part-to-part matching problem on a bipartite graph. The attention mechanism dynamically assigns lower weights to nodes corresponding to occluded regions, allowing the tracker to adapt to extreme deformations and partial blockages on the fly.
Similarly, the challenge of center-drift—where a tracker's bounding box shifts away from the true object center because it only focuses on the remaining visible fragment—is addressed using hierarchical networks. In CapsuleRRT: Relationships-Aware Regression Tracking via Capsules, Ding Ma et al. employ Capsule Networks to perform explicit part-to-whole reasoning. Because capsules encode both the presence of a feature and its spatial properties (such as orientation and position), the network can mathematically infer the location of the complete object based solely on the spatial orientation of its visible sub-parts.
By shifting from rigid, holistic templates to flexible, structurally constrained part representations, these tracking frameworks ensure that a system can reliably reason about the state of the whole object, even when it can only observe a fraction of it.
Key Foundational Work
Algebraic Set Kernels with Application to Inference Over Local Image Representations (2004) — contributed 1 concept elements to this direction.
Simultaneous Modeling and Tracking (SMAT) of Feature Sets (2005) — contributed 1 concept elements to this direction.