Mixed Supervised Object Detection with Robust Objectness Transfer

Yan Li, Junge Zhang, Kaiqi Huang, Jianguo Zhang

This paper, published in IEEE Transactions on Pattern Analysis and Machine Intelligence, presents a clever way to train object detectors when you have plenty of bounding box data for some categories but only image-level labels for others. By learning a "domain-invariant" sense of what an object looks like (objectness) and explicitly modeling distractors, the authors significantly bridge the performance gap between weakly supervised and fully supervised detection.

TL;DR

The authors propose Mixed Supervised Detection (MSD), a framework that transfers "objectness" knowledge from fully labeled categories to weakly labeled ones. By using domain adaptation to ensure this knowledge generalizes and a Multiple Instance Learning (MIL) strategy to filter out distractors, they achieve state-of-the-art results on major benchmarks like ILSVRC and PASCAL VOC.

What Problem Does This Paper Address?

Training high-performance object detectors (like Faster R-CNN) usually requires thousands of images with precise bounding boxes—a process that is expensive and slow. In contrast, image-level labels (e.g., "this image contains a cat") are easy to get.

The field of Weakly Supervised Object Detection (WSD) tries to learn detectors from these image-level labels alone. However, WSD has a major "pitfall": it often gets stuck on distractors. For example, a model might learn to identify a "cat face" or "water" instead of the "whole cat" or "boat" because those features are consistently present in the positive images.

This paper tackles Mixed Supervised Detection (MSD): How can we use a set of "strong" categories (with boxes) to help learn "weak" categories (with only image labels)? The challenge is that a detector trained on "dogs" doesn't automatically know how to find a "cat" or a "microwave" without falling into the same distractor traps.

Methods

The authors introduce a two-stage approach called Robust Objectness Transfer:

  1. Learning Domain-Invariant Objectness: Instead of transferring a specific category detector, they train a model to recognize "objectness"—the general "thing-ness" of an object regardless of its class. To ensure this generalizes from the strong categories (source) to the weak categories (target), they use a Domain Classifier with a Gradient Reversal Layer (GRL). This forces the neural network to learn features that make the two domains indistinguishable, ensuring the "objectness" model isn't biased toward the specific appearance of the strong categories.

  2. Objectness-aware Detection (MIL): Once they have a robust objectness scorer, they apply it to the weakly labeled images. They use the scores to sort region proposals:

  3. Object Bag: The top 15% of proposals likely to be objects.

  4. Distractor Bag: The remaining 85% likely to be background or parts.

    They then use a Multiple Instance Learning (MIL) framework. Unlike standard MIL, which only looks for the object, this version explicitly models the "distractor" concept. By training the network to distinguish between the "object bag" and the "distractor bag," the model learns to reject the very parts (like faces or context) that usually confuse weakly supervised models.

Datasets

The authors evaluated their method on several standard computer vision benchmarks:

  • ILSVRC2013: A large-scale dataset where they used 100 categories as "strong" and 100 as "weak."
  • PASCAL VOC 2007, 2010, and 2012: Used for both intra-dataset testing (splitting the 20 classes) and cross-dataset testing (transferring knowledge from ILSVRC to VOC).

Key Insights and Ideas

  • Objectness is a Universal Language: The most important "aha moment" is that while a "dog detector" is category-specific, the visual cues for "what constitutes an object" (edges, closed boundaries, contrast) are largely shared across categories.
  • The Power of Gradient Reversal: Simply training on strong categories leads to over-fitting on those specific shapes. By using an adversarial domain classifier, the model is forced to find the essence of an object that remains true across different datasets.
  • Learning What Isn't an Object: Most WSD methods fail because they are too "greedy"—they find the most discriminative part and stop. By explicitly creating a "distractor bag," this model learns that a "cat face" is a part of a cat, but not the whole cat, because it looks more like the items in the distractor bag than the items in the object bag.

Key Results

  • State-of-the-Art Performance: On the ILSVRC2013 detection task, their MSD approach achieved 25.26% mAP, more than doubling the performance of a standard weakly supervised baseline (11.82%).
  • Superior Transfer: In cross-dataset experiments (transferring from ILSVRC to PASCAL VOC 2007), their model reached 41.77% mAP (using AlexNet), significantly outperforming previous state-of-the-art transfer methods like LSSOD.
  • Robustness to Domain Shift: The method proved effective even when transferring from "man-made" objects to "natural" objects, showing that the learned objectness is truly robust.

Limitations and Future Work

Despite the gains, the authors note several remaining hurdles:

  • Complex Scenes: The model still struggles with categories like "chair" or "table" (see Figure 7). These objects often appear in clusters or overlap significantly, making it hard for the objectness scorer to pick out a single, complete instance.
  • Multiple Instances: In images with many people (e.g., a crowd), the MIL framework can struggle to separate individual instances, sometimes grouping them into one large box.
  • Future Directions: The authors suggest that incorporating more sophisticated context modeling or hierarchical object relationships could further improve the ability to distinguish between objects and their co-occurring distractors.

Summary updated

Abstract

In this paper, we consider the problem of leveraging existing fully labeled categories to improve the weakly supervised detection (WSD) of new object categories, which we refer to as mixed supervised detection (MSD). Different from previous MSD methods that directly transfer the pre-trained object detectors from existing categories to new categories, we propose a more reasonable and robust objectness transfer approach for MSD. In our framework, we first learn domain-invariant objectness knowledge from the existing fully labeled categories. The knowledge is modeled based on invariant features that are robust to the distribution discrepancy between the existing categories and new categories; therefore the resulting knowledge would generalize well to new categories and could assist detection models to reject distractors (e.g., object parts) in weakly labeled images of new categories. Under the guidance of learned objectness knowledge, we utilize multiple instance learning (MIL) to model the concepts of both objects and distractors and to further improve the ability of rejecting distractors in weakly labeled images. Our robust objectness transfer approach outperforms the existing MSD methods, and achieves state-of-the-art results on the challenging ILSVRC2013 detection dataset and the PASCAL VOC datasets.
BibTeX
@article{li2019mixed,
  author = {Yan Li and Junge Zhang and Kaiqi Huang and Jianguo Zhang},
  title = {{Mixed Supervised Object Detection with Robust Objectness Transfer}},
  journal = {IEEE Trans. Pattern Anal. Mach. Intell.},
  year = {2019},
  url = {https://openreview.net/forum?id=clKv3pTgXrl},
}

Loading related content...

Figure