This paper, published in IEEE Transactions on Pattern Analysis and Machine Intelligence, presents a clever way to train object detectors when you have plenty of bounding box data for some categories but only image-level labels for others. By learning a "domain-invariant" sense of what an object looks like (objectness) and explicitly modeling distractors, the authors significantly bridge the performance gap between weakly supervised and fully supervised detection.
TL;DR
The authors propose Mixed Supervised Detection (MSD), a framework that transfers "objectness" knowledge from fully labeled categories to weakly labeled ones. By using domain adaptation to ensure this knowledge generalizes and a Multiple Instance Learning (MIL) strategy to filter out distractors, they achieve state-of-the-art results on major benchmarks like ILSVRC and PASCAL VOC.
What Problem Does This Paper Address?
Training high-performance object detectors (like Faster R-CNN) usually requires thousands of images with precise bounding boxes—a process that is expensive and slow. In contrast, image-level labels (e.g., "this image contains a cat") are easy to get.
The field of Weakly Supervised Object Detection (WSD) tries to learn detectors from these image-level labels alone. However, WSD has a major "pitfall": it often gets stuck on distractors. For example, a model might learn to identify a "cat face" or "water" instead of the "whole cat" or "boat" because those features are consistently present in the positive images.
This paper tackles Mixed Supervised Detection (MSD): How can we use a set of "strong" categories (with boxes) to help learn "weak" categories (with only image labels)? The challenge is that a detector trained on "dogs" doesn't automatically know how to find a "cat" or a "microwave" without falling into the same distractor traps.
Methods
The authors introduce a two-stage approach called Robust Objectness Transfer:
-
Learning Domain-Invariant Objectness: Instead of transferring a specific category detector, they train a model to recognize "objectness"—the general "thing-ness" of an object regardless of its class. To ensure this generalizes from the strong categories (source) to the weak categories (target), they use a Domain Classifier with a Gradient Reversal Layer (GRL). This forces the neural network to learn features that make the two domains indistinguishable, ensuring the "objectness" model isn't biased toward the specific appearance of the strong categories.
-
Objectness-aware Detection (MIL): Once they have a robust objectness scorer, they apply it to the weakly labeled images. They use the scores to sort region proposals:
-
Object Bag: The top 15% of proposals likely to be objects.
-
Distractor Bag: The remaining 85% likely to be background or parts.
They then use a Multiple Instance Learning (MIL) framework. Unlike standard MIL, which only looks for the object, this version explicitly models the "distractor" concept. By training the network to distinguish between the "object bag" and the "distractor bag," the model learns to reject the very parts (like faces or context) that usually confuse weakly supervised models.
Datasets
The authors evaluated their method on several standard computer vision benchmarks:
- ILSVRC2013: A large-scale dataset where they used 100 categories as "strong" and 100 as "weak."
- PASCAL VOC 2007, 2010, and 2012: Used for both intra-dataset testing (splitting the 20 classes) and cross-dataset testing (transferring knowledge from ILSVRC to VOC).
Key Insights and Ideas
- Objectness is a Universal Language: The most important "aha moment" is that while a "dog detector" is category-specific, the visual cues for "what constitutes an object" (edges, closed boundaries, contrast) are largely shared across categories.
- The Power of Gradient Reversal: Simply training on strong categories leads to over-fitting on those specific shapes. By using an adversarial domain classifier, the model is forced to find the essence of an object that remains true across different datasets.
- Learning What Isn't an Object: Most WSD methods fail because they are too "greedy"—they find the most discriminative part and stop. By explicitly creating a "distractor bag," this model learns that a "cat face" is a part of a cat, but not the whole cat, because it looks more like the items in the distractor bag than the items in the object bag.
Key Results
- State-of-the-Art Performance: On the ILSVRC2013 detection task, their MSD approach achieved 25.26% mAP, more than doubling the performance of a standard weakly supervised baseline (11.82%).
- Superior Transfer: In cross-dataset experiments (transferring from ILSVRC to PASCAL VOC 2007), their model reached 41.77% mAP (using AlexNet), significantly outperforming previous state-of-the-art transfer methods like LSSOD.
- Robustness to Domain Shift: The method proved effective even when transferring from "man-made" objects to "natural" objects, showing that the learned objectness is truly robust.
Limitations and Future Work
Despite the gains, the authors note several remaining hurdles:
- Complex Scenes: The model still struggles with categories like "chair" or "table" (see Figure 7). These objects often appear in clusters or overlap significantly, making it hard for the objectness scorer to pick out a single, complete instance.
- Multiple Instances: In images with many people (e.g., a crowd), the MIL framework can struggle to separate individual instances, sometimes grouping them into one large box.
- Future Directions: The authors suggest that incorporating more sophisticated context modeling or hierarchical object relationships could further improve the ability to distinguish between objects and their co-occurring distractors.
Summary updated