Compositional Part Sharing Across Object Categories

108 concepts62 papers1 proposal

Traditional computer vision paradigms frequently treat object categories as isolated entities. If you train a model to recognize a horse, a cow, and a deer independently, the system typically learns redundant feature detectors for legs, eyes, and torsos three separate times. This independent strategy ignores the structural overlap inherent in the visual world, leading to severe computational redundancy and poor scaling as the number of target classes grows.

Cross-category part sharing flips this script. By treating the visual world as a compositional grammar built from a shared vocabulary of mid-level visual primitives, semantic parts, or structural components, models can represent thousands of diverse classes using a compact, unified dictionary. This approach provides two fundamental advantages: it forces sub-linear scaling in model complexity and enables rapid generalization to novel concepts.

Sub-Linear Scaling and Redundancy Reduction

The primary mathematical driver behind cross-category sharing is that the visual complexity of the world does not grow linearly with the number of categories. Early foundational work by Krystian Mikolajczyk demonstrated in Multiple Object Class Detection with a Generative Model that different object classes share a highly redundant visual vocabulary. Instead of maintaining independent templates for every viewpoint and category, clustering appearance features allows a single set of local descriptors to contribute to the detection of diverse classes.

When scaling up to handle hundreds or thousands of classes, this redundancy reduction prevents computational bottlenecks. Hans Lobel et al. demonstrated in Hierarchical Joint Max-Margin Learning of Mid and Top Level Representations for Visual Recognition that optimizing a shared dictionary across multiple categories avoids the inefficiency of binary one-vs-all classification schemes. By capturing cross-class correlations, a multi-class learning objective creates a far more concise feature set.

This sub-linear growth in vocabulary size as new categories are added was explicitly quantified by Ümit Rusen Aktas et al. in A Graph Theoretic Approach for Object Shape Representation in Compositional Hierarchies Using a Hybrid Generative-Descriptive Model, confirming that complex objects share substantial mid-level structural sub-structures. By reusing existing geometric primitives, the model minimizes the number of additional parameters required to integrate new classes.

Compositional Generalization in Low-Data Regimes

Beyond computational efficiency, cross-category sharing fundamentally changes how a model generalizes to novel concepts. In few-shot learning scenarios, data is too sparse to learn robust, class-specific features from scratch. However, if the model has already mastered a universal vocabulary of parts, recognizing a new object becomes a matter of composition rather than discovery. Learning a new class is simply a matter of learning a new spatial configuration of familiar parts.

Ruizhao Zhu et al. leveraged this intuition in Fine-grained Few-shot Recognition by Deep Object Parsing, using a shared dictionary of visual templates to describe novel classes through unique combinations and geometric arrangements of common elements. Similarly, Lihe Yang et al. showed in Mining Latent Classes for Few-shot Segmentation that decomposing object features into transferable sub-clusters provides a robust basis for meta-learning.

By encouraging inter-class compactness on the representation hypersphere rather than forcing maximum separation, the authors of MoTiC: momentum tightness and contrast for few-shot class-incremental learning enabled models to compositionally construct representations for novel classes using established base primitives. This data pooling means that a detector for a shared part (like a "wing") benefits from training data across birds, airplanes, and insects alike, drastically improving robustness.

Structural Mechanisms of Sharing

How do models mathematically enforce this sharing without losing their discriminative power? Several distinct mechanisms have emerged across dictionary learning, object detection, and semantic segmentation.

Shared Dictionary Learning

Traditional sparse coding often assigned independent dictionaries to individual classes. Julien Mairal showed in Supervised Dictionary Learning that this approach is highly inefficient; instead, a single shared dictionary trained across all categories allows the model to learn universal structural primitives (like edges or textures) while dedicating specific atoms to unique class traits.

NAVEED AKHTAR et al. expanded on this in Discriminative Bayesian Dictionary Learning for Classification, noting that a rigid one-to-one mapping between atoms and classes ignores natural visual overlap, whereas shared dictionaries with automatic sub-dictionary discovery permit flexible feature sharing that adapts to the inherent overlap across categories.

Super-Class Filter Reuse

In large-scale object detection, maintaining unique localization filters for thousands of classes is a massive resource drain. Bharat Singh et al. solved this bottleneck in R-FCN-3000 at 30fps: Decoupling Detection and Classification by clustering visually similar object categories into semantic "super-classes." This architectural decoupling allows localization filters to be shared across different types of objects, reducing the total number of position-sensitive filters required for accurate bounding box prediction.

Semantic Compositional Parts and Prototypes

For fine-grained localization and segmentation, Peng Wang et al. introduced a joint framework in Joint Object and Part Segmentation Using Deep Learned Potentials that groups visually similar components across different object classes into a unified semantic label space (e.g., mapping limbs across different animal species to the same underlying representation).

In self-explainable models, Dawid Rymarczyk demonstrated in Interpretable Image Classification with Differentiable Prototypes Assignment that sharing a common library of visual prototypes across classes achieves a ten-fold decrease in the number of required prototypes while maintaining high accuracy.

Finally, Wanli Ouyang showed in DeepID-Net: Deformable deep convolutional neural networks for object detection that jointly learning shared visual patterns alongside class-specific deformation constraints allows models to recognize common structural elements (such as circular patterns found in both wheels and musical instruments) across diverse categories.

History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 67 papers
  2. Version 1 · 2026-10-04 · 116 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_cross_category_part_sharing_dd309052,
  title = {cross-category part sharing},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/cross-category-part-sharing/rdr_dd309052d7f941da9c02322e0405b6c3},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 2, frozen text):

@misc{lacuna_cross_category_part_sharing_dd309052_v2,
  title = {cross-category part sharing},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/cross-category-part-sharing/rdr_dd309052d7f941da9c02322e0405b6c3/v/2},
  note = {Version 2, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake