Part-Based Object Representations for Occlusion-Robust Vision

129 concepts97 papers

When a computer vision system attempts to track or recognize an object, its standard instinct is to compress the target into a single, holistic representation—a global bounding box, a template, or a high-dimensional feature vector. While computationally convenient, this holistic strategy is fundamentally brittle. If a pedestrian walks behind a mailbox, or a vehicle is partially obscured by a tree, the global visual signature changes drastically. For a monolithic model, this localized corruption causes a catastrophic failure: the entire feature vector collapses, and the tracker or classifier loses its lock.

Part-based object representation addresses this vulnerability by shifting from a holistic paradigm to a distributed one. Instead of modeling an object as a single rigid entity, these techniques decompose it into a collection, grid, or hierarchy of localized sub-components.

The core intuition behind why this works is graceful degradation. If an object is represented as twenty independent local parts, an occlusion that hides 30% of the object will completely corrupt six parts, but leave the remaining fourteen perfectly intact. By aggregating evidence from the unaffected regions, the system can maintain a stable track, estimate 3D pose, or confirm identity despite severe spatial interference.


The Geometry of Decomposition

To build a part-based representation, a system must first decide how to break an object down. Across the literature, three primary geometric strategies emerge: rigid patches, structural strips, and adaptive superpixels.

1. Uniform and Overlapping Patches

The simplest way to decompose an image is to divide it into a grid of localized tiles. Early foundations laid by Chris Bregler et al. in Learning Appearance Based Models: Mixtures of Second Moment Experts demonstrated that breaking an image into local overlapping tiles significantly reduces sensitivity to global pose variations and occlusions.

In modern architectures, this manifests as patch-based feature extraction. For instance, Fan Liu et al. showed in Patch based Semi-supervised Linear Regression for Face Recognition that analyzing overlapping local patches captures fine-grained facial variations (like lighting and expression) far better than global linear regression. When dealing with highly non-rigid targets, Mingzhen Huang et al. demonstrated in Tracking Multiple Deformable Objects in Egocentric Videos that representing objects as a grid of flexible patches instead of rigid bounding boxes allows trackers to maintain a consistent lock even when the target twists, stretches, or is partially obscured.

2. Anatomical and Structural Strips

For specific object classes like humans or vehicles, unaligned grids are less efficient than domain-specific structural divisions. In person re-identification (Re-ID) tasks, researchers frequently slice feature maps into horizontal strips to isolate body regions.

Jiayun Wang et al. leveraged this in Deep Ranking Model by Large Adaptive Margin Learning for Person Re-identification, fusing global context with localized horizontal patches to ensure the network remains robust when specific body segments are hidden. Yang Fu et al. extended this concept to a multi-scale architecture in Horizontal Pyramid Matching for Person Re-Identification, creating a redundant representation where every horizontal segment is trained to be independently discriminative.

Outside of deep learning feature maps, explicit anatomical partitioning yields similar benefits. Hatice Gunes demonstrated in Tracking People in Crowds by a Part Matching Approach that breaking the human body into five distinct segments and using the head as a reliable "anchor" allows trackers to successfully navigate long-term, repeated occlusions in crowded environments.

3. Adaptive Regions and Superpixels

Rigid grids and strips suffer from a shared flaw: they do not conform to the actual physical boundaries of an object, meaning a single patch often contains both target pixels and background noise. To solve this, trackers utilize mid-level features like superpixels or supervoxels.

As shown by Sheng Chen et al. in Multi-object Tracking via Constrained Sequential Labeling, tracking dense point trajectories and supervoxels instead of high-level bounding boxes provides a tighter fit around physical object boundaries. Similarly, Hirokatsu Kataoka et al. demonstrated in People Tracking in Dense Crowds using Optical Flow Clustering based Superpixel that grouping optical flow data into adaptive superpixels creates a spatial middle ground that conforms to an object's shifting silhouette, preventing the tracker from drifting onto background clutter during partial occlusions.


Why it Works: The Mechanics of Evidence Aggregation

Decomposing an object into parts is only half the battle; the system must also know how to reason about these parts collectively without re-introducing global brittleness. The literature highlights three core mechanisms for robust part aggregation.

Independent Classifier Committees

Instead of feeding all parts into a single classifier, systems often treat individual parts as independent experts. Massimiliano Mancini et al. implemented this by embedding a Naïve Bayes Nearest Neighbor (NBNN) classifier into a fully-convolutional network in Learning Deep NBNN Representations for Robust Place Categorization. By classifying an image as a loose collection of local descriptors rather than a single global vector, their model maintains high accuracy even when large portions of the scene change or are obstructed.

For fine-grained recognition, Upal Mahbub showed in Segment-based Methods for Facial Attribute Detection from Partial Faces that dividing a face into 14 distinct segments and processing them through a "committee" of specialized neural networks ensures that if a subject wears sunglasses or a mask, the networks covering the unoccluded regions (like the mouth or jawline) still deliver highly accurate attribute predictions.

Holistic Paradigm:
[ Image ] ---> [ Global Feature Vector ] ---> [ Single Classifier ] ---> (Fails if occluded)

Part-Based Paradigm:
               /---> [ Patch 1 ] ---> [ Expert Filter 1 ] ---\
[ Image ] ---> ----> [ Patch 2 ] ---> [ Expert Filter 2 ] ----> [ Evidence Aggregation ] ---> Robust Output
               \---> [ Patch 3 ] ---> [ Expert Filter 3 ] ---/

Distributed Tracking Filters

In visual tracking, running multiple independent trackers simultaneously provides a safety net. Ting Liu 0009 et al. operationalized this in Real-time part-based visual tracking via adaptive correlation filters by assigning a dedicated correlation filter to multiple independent sub-regions. Because these filters operate efficiently in the Fourier domain, the system maintains real-time speeds while dynamically lowering the weight of any sub-region whose confidence map drops due to occlusion.

This decoupled approach is highly flexible; as José Miguel Buenaposada et al. noted in Efficient Appearance-Based Tracking, different parts of an object can maintain entirely independent appearance models while sharing a unified motion model, allowing the system to track complex, deforming targets without losing structural coherence.

Generative Compositionality

While discriminative models try to map features directly to labels, generative part-based models learn the underlying spatial distribution of parts. Angtian Wang demonstrated the power of this approach in Robust Object Detection Under Occlusion With Context-Aware CompositionalNets. By modeling objects as generative spatial collections of parts, CompositionalNets can detect an object as long as a critical subset of its parts matches the generative model, completely bypassing the context-reliance issues that cause standard detectors like Faster R-CNN to fail under heavy occlusion.


Beyond Bags of Parts: Graphs and Hierarchies

Treating an object as an un-ordered "bag of parts" introduces a new problem: it discards spatial context. A face model that accepts two eyes, a nose, and a mouth regardless of their arrangement would falsely recognize a scrambled face. To maintain robustness without sacrificing spatial logic, modern research structures parts using graphs, hierarchies, and attention slots.

Graph-Based Part Matching

By treating local parts as nodes and their spatial or semantic relationships as edges, objects can be modeled as topological networks. Yijuan Lu utilized this in RGB-T Object Tracking: Benchmark and Baseline, representing image patches as nodes in a weighted graph to dynamically balance and fuse visible and thermal data streams.

When an object undergoes extreme deformation, rigid templates fail completely. To combat this, Dongyan Guo et al. introduced SiamGAT in Graph Attention Tracking. By reframing tracking as a part-to-part matching problem on a bipartite graph, their model uses a Graph Attention Module to dynamically compute similarity between target parts and search patches, allowing the tracker to adapt to drastic aspect ratio changes and non-rigid warping.

Hierarchical and Slot-Based Architectures

Objects are naturally compositional—wheels belong to cars, which belong to traffic scenes. Capturing this multi-scale reality requires hierarchical modeling. Mete Özay formalized this in Compositional Hierarchical Representation of Shape Manifolds for Classification of Non-manifold Shapes, breaking down complex, disconnected shapes into a tree-like vocabulary of simpler manifold parts. This structural composition proved significantly more resilient to background noise than standard flat architectures.

In multi-object tracking, Jiangmiao Pang et al. developed the CSC-Tracker in Multi-Object Tracking by Hierarchical Visual Representations, which simultaneously extracts features across a three-level hierarchy: Compositional (parts), Semantic (the whole object), and Contextual (the environment). This multi-tiered focus ensures that if the semantic bounding box becomes noisy or overlaps with another target, the lower-level compositional part embeddings preserve the object's unique identity.

Hierarchical Representation:
       [ Global Object: Pedestrian ]
               /           \
   [ Upper Body ]         [ Lower Body ]
      /      \               /      \
 [ Head ]  [ Torso ]    [ Left Leg ] [ Right Leg ]

The frontier of this research focuses on discovering these hierarchies without manual annotation. Jiahao Xia et al. achieved this in Unsupervised Part Discovery via Dual Representation Alignment, using a specialized transformer module to automatically decompose objects into semantically consistent, localized part embeddings.

Similarly, in cross-modal tasks, Jicheol Park et al. introduced the PLOT framework in PLOT: Text-Based Person Search with Part Slot Attention for Corresponding Part Discovery. PLOT utilizes a slot attention mechanism where discrete, bounded slots compete to represent specific regions of an image. When a textual query describes a person ("a man wearing a red backpack"), the model doesn't look at the global image; instead, it dynamically weights the specific slot corresponding to the torso and back, isolating the relevant part and ignoring any occlusions or viewpoint changes affecting the rest of the subject.

Key Foundational Work

An Approach to Image Retrieval for Image Databases (1993) — contributed 2 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_part_based_object_representation_1659287a,
  title = {part-based object representation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/part-based-object-representation/rdr_1659287a79634b41a2cef2e02c5ee415},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_part_based_object_representation_1659287a_v1,
  title = {part-based object representation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/part-based-object-representation/rdr_1659287a79634b41a2cef2e02c5ee415/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}

See something wrong? Report a mistake