TL;DR
Jointly detecting objects and predicting their attributes (like color and material) typically degrades detection performance because the two tasks require conflicting feature representations. This paper introduces a two-stream architecture that disentangles object and attribute features, resolving this conflict. The proposed model not only recovers the lost object detection accuracy but also significantly improves attribute recognition, especially when transferring attributes to novel object categories.
What Problem Does This Paper Address?
In computer vision, we often want models to do more than just draw a box around a "car"—we want to know it's a "red, metal car." However, adding attribute prediction heads to a standard object detector (like Faster R-CNN) creates an internal conflict.
Object detection needs to be attribute-independent (a car is a car, regardless of its color). Conversely, attribute recognition needs to be largely object-independent (red is red, whether it's on a car or a shirt). When a single neural network backbone tries to learn features for both tasks simultaneously, these features become "entangled." The authors observe that this entanglement causes a massive drop in basic object detection accuracy.
Furthermore, the paper addresses the challenge of attribute transfer. Real-world datasets cannot possibly contain every combination of object and attribute. A robust model must be able to learn what "red" looks like on a shirt and successfully identify a "red car" during testing, even if it has never seen a red car during training.
Methods
To eliminate feature entanglement, the authors propose a Two-Stream Architecture built on top of the R-CNN pipeline:
- Shared Proposals, Split Backbones: The model uses a shared Region Proposal Network (RPN) to find regions of interest (RoIs). However, instead of passing these RoIs through a single feature extractor, they are fed into two independent ResNet-101 streams: an Object Stream and an Attribute Stream.
- The Cross Link: While color is mostly independent of object category, material is highly correlated with it (e.g., windows are usually glass). To leverage this, the authors add a one-way "cross link" that passes features from the Object Stream into the Attribute Stream just before prediction.
- Stop-Gradient: Crucially, they block the gradient from flowing backward through this cross link. This allows the attribute predictor to use object context without forcing the object detector to update its weights based on attribute errors, preserving detection accuracy.
- Separated Cross-Entropy (SCE) Loss: Instead of treating all attributes as a single flat list, they compute separate loss functions for color and material.
Datasets
Because standard detection datasets lack attribute annotations, and datasets like Visual Genome suffer from severe long-tail distributions and overlapping categories, the authors created a curated subset called VG-20.
- VG-20 consists of 20 object categories, 12 descriptive colors, and 4 common materials.
- It contains 180k training samples and 25k testing samples.
- The dataset is evaluated under two splits: a standard Supervised split, and an Attribute Transfer split (where objects are divided into reference and target groups, and target groups have their attribute labels hidden during training to test zero-shot transfer).
Key Insights and Ideas
- Feature entanglement is a severe bottleneck: Naively combining object detection and attribute recognition in a single-stream model drops object detection mean Average Precision (mAP) by over 10%. Disentangling them is strictly necessary for performant joint models.
- Information sharing should be asymmetric: While object features shouldn't care about attributes, attribute features do benefit from knowing the object. The one-way cross-link perfectly captures this asymmetry, boosting attribute recall without penalizing the detector.
- Late-stage entanglement still fails: The authors tested a variant ("Late Feature Entanglement") where features were kept separate in the backbones but concatenated right before the final classifiers. This still caused a ~10% drop in mAP, proving that separation must be maintained entirely through the classification heads.
Key Results
- Supervised Setting: A baseline detection-only model achieved 38.18% mAP. Adding attribute heads to a single-stream model tanked this to 25.13%. The proposed Two-Stream + Cross Link model fully recovered the detection performance (38.30% mAP) while boosting color recall to 73.11% (up from 68.59%) and material recall to 65.39% (up from 61.22%).
- Attribute Transfer Setting: On unseen object-attribute pairings, the Two-Stream + Cross Link model proved vastly superior. It achieved 38.39% mAP and 61.01% color recall on target categories, crushing the baseline single-stream model (which managed only 24.87% mAP and 48.39% color recall).
Limitations and Future Work
The authors do not include a dedicated "Future Work" section, but they do acknowledge inherent limitations in the problem space. Specifically, using rectangular bounding boxes inevitably mixes background pixels with object pixels, which complicates color and material recognition. Additionally, while VG-20 cleanly demonstrates the entanglement problem, it is a highly constrained subset (only 20 objects and 16 total attributes). An implicit future direction would be scaling this two-stream, cross-linked architecture to the massive, noisy, and long-tailed vocabularies found in full scene-graph generation tasks.
Summary updated