Mixed Supervised Object Detection with Robust Objectness Transfer

Yan Li, Junge Zhang, Jianguo Zhang, Kaiqi Huang

Abstract: In this paper, we consider the problem of leveraging existing fully labeled categories to improve the weakly supervised detection (WSD) of new object categories, which we refer to as mixed supervised detection (MSD). Different from previous MSD methods that directly transfer the pre-trained object detectors from existing categories to new categories, we propose a more reasonable and robust objectness transfer approach for MSD. In our framework, we first learn domain-invariant objectness knowledge from the existing fully labeled categories. The knowledge is modeled based on invariant features that are robust to the distribution discrepancy between the existing categories and new categories; therefore the resulting knowledge would generalize well to new categories and could assist detection models to reject distractors (e.g., object parts) in weakly labeled images of new categories. Under the guidance of learned objectness knowledge, we utilize multiple instance learning (MIL) to model the concepts of both objects and distractors and to further improve the ability of rejecting distractors in weakly labeled images. Our robust objectness transfer approach outperforms the existing MSD methods, and achieves state-of-the-art results on the challenging ILSVRC2013 detection dataset and the PASCAL VOC datasets.

Introduction

Training high-performance object detectors usually requires thousands of images with precise bounding boxes—a process that is expensive and slow. In contrast, image-level labels (e.g., "this image contains a cat") are easy to obtain. The field of Weakly Supervised Object Detection (WSD) attempts to learn detectors from these image-level labels alone. However, WSD often suffers from a major pitfall: it gets stuck on distractors. For example, a model might learn to identify a "cat face" or "water" instead of the "whole cat" or "boat" because those features are consistently present in the positive images.

Illustrating the distractor trap in WSD: models often focus on discriminative parts (red) rather than the whole object (green).

This paper addresses Mixed Supervised Detection (MSD): leveraging a set of "strong" categories (with bounding boxes) to help learn "weak" categories (with only image labels). The challenge is ensuring that a detector trained on specific source categories generalizes to new target categories without falling into the same distractor traps.

Proposed Method: Robust Objectness Transfer

The authors introduce a two-stage approach to bridge the gap between strong and weak supervision:

1. Learning Domain-Invariant Objectness

Instead of transferring a specific category detector, the framework trains a model to recognize "objectness"—the general "thing-ness" of an object regardless of its class. To ensure this generalizes from strong categories (source) to weak categories (target), the authors employ a Domain Classifier with a Gradient Reversal Layer (GRL). This adversarial setup forces the neural network to learn features that make the two domains indistinguishable, ensuring the objectness model isn't biased toward the specific appearance of the strong categories.

2. Objectness-aware Multiple Instance Learning (MIL)

Once a robust objectness scorer is established, it is applied to weakly labeled images to sort region proposals into two groups:

  • Object Bag: Proposals with high objectness scores (top 15%), likely to contain whole objects.
  • Distractor Bag: The remaining proposals, likely to be background or object parts.
The proposed Mixed Supervised Detection (MSD) framework, featuring domain-invariant objectness transfer and MIL-based modeling.

Unlike standard MIL, which only seeks the object, this version explicitly models the "distractor" concept. By training the network to distinguish between the "object bag" and the "distractor bag," the model learns to reject the very parts (like faces or context) that typically confuse weakly supervised models.

Experimental Results

The authors evaluated their method on several standard computer vision benchmarks, including ILSVRC2013 and PASCAL VOC 2007, 2010, and 2012.

  • State-of-the-Art Performance: On the ILSVRC2013 detection task, the MSD approach achieved 25.26% mAP, more than doubling the performance of a standard weakly supervised baseline (11.82%).
  • Superior Transfer: In cross-dataset experiments (transferring from ILSVRC to PASCAL VOC 2007), the model reached 41.77% mAP (using AlexNet), significantly outperforming previous state-of-the-art transfer methods.
  • Robustness to Domain Shift: The method proved effective even when transferring from "man-made" objects to "natural" objects, demonstrating that the learned objectness is truly robust across different visual domains.

Conclusion and Future Work

The research demonstrates that "objectness" is a universal language; while detectors are category-specific, the visual cues for what constitutes an object are largely shared. By explicitly modeling what isn't a whole object (distractors), the framework significantly improves detection accuracy.

Despite these gains, challenges remain in complex scenes where objects like "chairs" or "tables" appear in dense clusters or overlap significantly. Future work may involve incorporating more sophisticated context modeling or hierarchical object relationships to further refine the distinction between objects and their co-occurring distractors.

Summary updated

Abstract

In this paper, we consider the problem of leveraging existing fully labeled categories to improve the weakly supervised detection (WSD) of new object categories, which we refer to as mixed supervised detection (MSD). Different from previous MSD methods that directly transfer the pre-trained object detectors from existing categories to new categories, we propose a more reasonable and robust objectness transfer approach for MSD. In our framework, we first learn domain-invariant objectness knowledge from the existing fully labeled categories. The knowledge is modeled based on invariant features that are robust to the distribution discrepancy between the existing categories and new categories; therefore the resulting knowledge would generalize well to new categories and could assist detection models to reject distractors (e.g., object parts) in weakly labeled images of new categories. Under the guidance of learned objectness knowledge, we utilize multiple instance learning (MIL) to model the concepts of both objects and distractors and to further improve the ability of rejecting distractors in weakly labeled images. Our robust objectness transfer approach outperforms the existing MSD methods, and achieves state-of-the-art results on the challenging ILSVRC2013 detection dataset and the PASCAL VOC datasets.
BibTeX
@article{li2019mixed,
  author = {Yan Li and Junge Zhang and Jianguo Zhang and Kaiqi Huang},
  title = {{Mixed Supervised Object Detection with Robust Objectness Transfer}},
  journal = {IEEE Trans. Pattern Anal. Mach. Intell.},
  year = {2019},
  url = {https://openreview.net/forum?id=clKv3pTgXrl},
}
Figure