CS-R-FCN: Cross-Supervised Learning for Large-Scale Object Detection

Ye Guo, Yali Li, Shengjin WangICASSP 2020 · DOI

TL;DR

This paper introduces CS-R-FCN, a novel cross-supervised learning framework for large-scale object detection. It cleverly leverages both bounding-box and image-level annotations to train a two-stage detector, significantly improving performance over prior methods by introducing a semantic aggregation strategy to handle category confusion.

What Problem Does This Paper Address?

Object detection, the task of identifying and localizing objects in images, is a fundamental computer vision problem. However, scaling this to detect thousands of categories (large-scale object detection) presents a significant challenge, primarily due to the immense cost and effort required to obtain bounding-box-level annotations for such a vast number of classes. Existing methods struggle to effectively utilize readily available image-level annotations (which only indicate the presence of an object, not its precise location) in conjunction with the more informative but scarce bounding-box annotations, especially within efficient, end-to-end trainable frameworks. This paper aims to bridge this gap by developing a cross-supervised learning approach that maximizes the utility of both annotation types for large-scale object detection.

Methods

The core of CS-R-FCN is a novel cross-supervised learning pipeline built upon the R-FCN-3000 two-stage object detection framework. The key innovations are:

  • Cross-Supervised Learning Pipeline: The framework integrates image-level annotated data into the region proposal network (RPN) and the classification branch of the two-stage detector. While bounding-box annotations are used for standard training of the RPN and detection branch, image-level annotations are used to finetune the classification branch. Proposals generated by the RPN are assigned positive samples for corresponding categories from the image-level data.
  • CS-RPN (Cross-Supervised Region Proposal Network): A modified RPN that can be trained using both bounding-box and image-level annotated data. This allows for the generation of more robust and relevant proposals for the subsequent detection and classification stages.
  • Semantic Aggregation Strategy: To mitigate the "mutual inhibition effects" that arise when training with many categories, especially in a hierarchical manner, the authors propose a semantic aggregation strategy. This involves building a hierarchical structure of categories (using WordNet) and focusing the classification loss on leaf nodes. Ancestor category probabilities are then aggregated from their descendant leaf nodes during inference. This reduces conflicts and improves learning efficiency.

Datasets

  • Training Data: A combined dataset consisting of the ILSVRC DET training set (bounding-box-level annotations for 544 categories) and the ILSVRC CLS-LOC training set (image-level annotations for 506 categories).
  • Evaluation Data:
    • ILSVRC DET validation set (200 categories).
  • ILSVRC CLS-LOC validation hard set (1,000 categories, with images containing multiple objects) for a more rigorous evaluation.

Key Insights and Ideas

  • Leveraging Image-Level Data for Proposals: The paper demonstrates that image-level annotations, often considered less informative for localization, can still contribute to generating better region proposals when integrated into the RPN. This is a crucial step for enabling cross-supervised learning in two-stage detectors.
  • The Challenge of Category Confusion: For large-scale detection, the sheer number of categories leads to complex relationships and potential confusion. The proposed semantic aggregation directly addresses this by organizing categories hierarchically and focusing learning on more specific (leaf) categories, reducing interference.
  • Efficiency of Hierarchical Classification: By focusing on leaf nodes and aggregating probabilities, the authors achieve a more efficient hierarchical classification compared to methods like YOLO-9000, which might require more complex tree structures and multiple softmax operations.

Key Results

  • Improved mAP: The proposed CS-R-FCN with semantic aggregation achieves a mean Average Precision (mAP) of 36.2% on the ILSVRC DET validation set.
  • Significant Gains from Semantic Aggregation: Without semantic aggregation, CS-R-FCN achieved 31.6% mAP. The semantic aggregation strategy improved this by 4.6%.
  • Competitive Performance: CS-R-FCN (w/ SA) achieves comparable performance to the fully-supervised baseline R-FCN-3000, despite using significantly less bounding-box-level annotated data.
  • Outperforming Prior Cross-Supervised Methods: CS-R-FCN shows substantial improvements over LSDA (14.1% mAP gain) and YOLO-9000 (16.5% mAP gain).
  • Proposal Quality: The CS-RPN demonstrates good proposal generation capabilities, achieving an AP of 63.4% for bounding-box-level categories and 47.6% for image-level categories with just 10 proposals.

Limitations and Future Work

  • Limitations: The paper acknowledges that while CS-R-FCN improves upon previous methods, there's still room for further enhancement. The semantic aggregation strategy, while effective, relies on an external hierarchical structure (WordNet), which might not perfectly capture all semantic relationships relevant to the specific dataset.
  • Future Work: The authors suggest that their proposed cross-supervised learning pipeline, particularly the semantic aggregation strategy, can be further optimized. They also imply that extending this approach to even larger-scale detection tasks and exploring more sophisticated ways to model category relationships could be fruitful directions.

Summary updated

BibTeX
@inproceedings{guo2020cs,
  author = {Ye Guo and Yali Li and Shengjin Wang},
  title = {{CS-R-FCN: Cross-Supervised Learning for Large-Scale Object Detection}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2020},
  doi = {10.1109/icassp40776.2020.9053689},
}
Figure