Abstract
We explore object detection with two attributes: color and material. The task aims to simultaneously detect objects and infer their color and material. A straight-forward approach is to add attribute heads at the very end of a usual object detection pipeline. However, we observe that the two goals are in conflict: Object detection should be attribute-independent and attributes be largely object-independent. Features computed by a standard detection network entangle the category and attribute features; we disentangle them by the use of a two-stream model where the category and attribute features are computed independently but the classification heads share Regions of Interest (RoIs). Compared with a traditional single-stream model, our model shows significant improvements over VG-20, a subset of Visual Genome, on both supervised and attribute transfer tasks.
The Challenge of Feature Entanglement
In computer vision, the goal is often to move beyond simple object detection toward a richer understanding of scene components, such as identifying a "red, metal car" rather than just a "car." However, naively adding attribute prediction heads to standard detectors like Faster R-CNN introduces an internal conflict. Object detection requires features that are attribute-independent (a car's identity is constant regardless of color), while attribute recognition requires features that are largely object-independent (the visual signature of "red" should be consistent across shirts and cars).
When a single backbone attempts to learn both, features become "entangled," leading to a significant drop in detection performance—often exceeding 10% mAP. Furthermore, models must handle attribute transfer: the ability to recognize an attribute on a novel object category (e.g., learning "red" from a shirt to identify a red car) even if that specific combination was never seen during training.
The Two-Stream Architecture
To resolve this conflict, we propose a Two-Stream Architecture that disentangles object and attribute features while allowing for necessary information sharing.
Key Components:
- Shared Proposals, Split Backbones: The model utilizes a shared Region Proposal Network (RPN) to identify Regions of Interest (RoIs). These RoIs are then processed by two independent ResNet-101 streams: an Object Stream and an Attribute Stream.
- Asymmetric Cross-Link: While object detection should ignore attributes, attribute recognition often benefits from object context (e.g., knowing an object is a window helps identify its material as glass). We implement a one-way cross-link that passes features from the Object Stream to the Attribute Stream.
- Stop-Gradient: To prevent attribute-related errors from corrupting the object detector, we block gradients from flowing back through the cross-link. This ensures the detector remains focused on category-invariant features.
- Separated Cross-Entropy (SCE) Loss: Attributes are grouped (e.g., color vs. material) and optimized using separate loss functions rather than a single flat classification head.
Experimental Evaluation
We evaluated our approach on VG-20, a curated subset of the Visual Genome dataset containing 20 object categories, 12 colors, and 4 materials.
Supervised Learning
In a standard supervised setting, a single-stream model's detection performance dropped from 38.18% to 25.13% mAP when attribute heads were added. Our Two-Stream + Cross Link model fully recovered this performance (38.30% mAP) while simultaneously improving color recall (73.11% vs 68.59%) and material recall (65.39% vs 61.22%).
Attribute Transfer
The advantages were even more pronounced in the transfer setting. When tested on unseen object-attribute pairings, our model achieved 38.39% mAP and 61.01% color recall, significantly outperforming the single-stream baseline (24.87% mAP and 48.39% color recall).
Conclusion
Feature entanglement is a major bottleneck in joint object-attribute tasks. By utilizing a two-stream architecture with asymmetric information sharing and gradient blocking, we can successfully disentangle these tasks. This approach not only preserves detection accuracy but also enhances the model's ability to generalize attributes to novel objects, a critical requirement for robust real-world vision systems.
Summary updated