Co-occurrence matrix analysis-based semi-supervised training for object detection

Min-Kook Choi, Jaehyeong Park, Jihun Jung, Heechul Jung, Jin-Hee Lee, Woong Jae Won, Woo Young Jung, Jincheol Kim, Soon Kwon

TL;DR

This paper introduces a semi-supervised learning (SSL) framework for object detection that improves accuracy without adding model parameters or altering network architectures. By filtering and refining automatically generated "pseudo-labels" using a co-occurrence matrix—which captures how likely different objects are to appear together in an image—the method successfully leverages unlabeled data to boost the performance of popular detectors like YOLO v2, SSD, and D-ConvNets.

What Problem Does This Paper Address?

Deep learning-based object detectors thrive on massive amounts of labeled data, but human annotation for bounding boxes is incredibly time-consuming and expensive. Semi-supervised learning (SSL) attempts to solve this by training a model on a small labeled dataset, using it to predict labels (pseudo-labels) on a large unlabeled dataset, and then retraining the model on the combined data.

However, standard pseudo-labeling has a major flaw: the generated labels are heavily dependent on the biases and parameters of the initial model. If the model makes a confident but incorrect prediction, retraining on that "error" just reinforces the mistake, yielding little to no performance gain. The authors tackle this bottleneck by introducing a realignment method that validates these pseudo-labels using real-world object relationships.

Methods

The authors propose a two-phase SSL pipeline:

  1. Phase 1 (Initial Training & Inference): An object detector is trained on standard annotated data. This model then runs inference on an unlabeled dataset. Predictions with a softmax confidence score above a threshold ($\rho$) are converted into one-hot encoded pseudo-labels.
  2. Co-occurrence Matrix Realignment: To filter out bad pseudo-labels, the authors introduce a prior-knowledge filter based on object co-occurrence (how often classes like 'dog' and 'horse' appear in the same image). They construct a co-occurrence matrix from the training set, apply conditional marginalization, and perform max-normalization to balance out dominant classes. When multiple objects are detected, the framework adjusts the confidence of a pseudo-label based on its correlation with the most dominant companion object in the scene.
  3. Phase 2 (Retraining): The network is retrained from scratch using both the original annotated data and the newly filtered, high-confidence pseudo-labeled data.
Fig. 1: Overview of the proposed SSL pipeline. The pro- posed technique consists of two steps. In the first step, the detector for labeling the unlabeled data is trained with the ex- isting annotated data (Training1), and then the inferring pro- cess for the unlabeled data is performed (Testing1). After performing pseudo-labeling through one-hot-vector encoding and co-occurrence matrix analysis, a new network is trained (Training2) and the data for evaluation are inferred (Testing2).

Datasets

The method was evaluated using the MS-COCO 2017 detection dataset, utilizing several splits:

  • train17 + val17 for initial supervised training.
  • unlabeled17 containing unlabeled images to test the semi-supervised pipeline.
  • test-dev17 for final quantitative evaluation.

Key Insights and Ideas

  • Exploiting Semantic Context: Instead of treating object detections in an image as independent events, the framework leverages the fact that real-world scenes have predictable structures (e.g., a 'tie' is highly correlated with a 'person').
  • Max-Normalization is Crucial: Raw co-occurrence counts are heavily biased toward frequently occurring classes. By applying max-normalization, the authors prevent common objects from completely drowning out rarer, highly specific object correlations.
  • The "Two Strongest Objects" Rule: To keep computation efficient and robust, when multiple objects appear, the correction mechanism simplifies the joint probability by focusing primarily on the relationship between the target object and the single most confident companion object in the image.
Fig. 2: Example of co-occurrence matrix. A co-occurrence matrix of four classes with five images (top) and a case with prior knowledge from a large-scale dataset (bottom) using the conditional marginalization and the max-normalization.

Key Results

The proposed SSL method consistently improved performance across different object detection architectures on the MS-COCO dataset without modifying their underlying structures:

  • Deformable Convolutional Networks (D-ConvNets): Baseline mean Average Precision (mAP) improved from 36.3% to 37.8% (a +1.5% absolute gain) under the optimal thresholds ($\rho = 0.5, \rho_{co} = 0.3$).
  • SSD (Single Shot MultiBox Detector): Performance increased from 24.1% to 25.3% mAP.
  • YOLO v2: Performance increased from 24.0% to 25.1% mAP.

Qualitatively, the co-occurrence filter successfully removed false positives (mis-inferred objects) and helped recover small or heavily occluded objects that the baseline model initially missed by utilizing the context of surrounding objects.

Limitations and Future Work

The authors note a few areas for improvement and further study:

  • Increased Training Time: Because the volume of training data expands significantly with the inclusion of unlabeled images, the overall training time increases accordingly.
  • Hyperparameter Sensitivity: The performance relies on tuning multiple thresholds ($\rho$ and $\rho_{co}$), which might vary depending on the dataset used.
  • Architecture Impact: Future work is needed to analyze how this SSL scheme behaves across different network types (specifically comparing its impact on one-stage vs. two-stage detectors) and to explore the exact dependency on the baseline detector's initial strength.

Summary updated

Abstract

One of the most important factors in training object recognition networks using convolutional neural networks (CNNs) is the provision of annotated data accompanying human judgment. Particularly, in object detection or semantic segmentation, the annotation process requires considerable human effort. In this paper, we propose a semi-supervised learning (SSL)-based training methodology for object detection, which makes use of automatic labeling of un-annotated data by applying a network previously trained from an annotated dataset. Because an inferred label by the trained network is dependent on the learned parameters, it is often meaningless for re-training the network. To transfer a valuable inferred label to the unlabeled data, we propose a re-alignment method based on co-occurrence matrix analysis that takes into account one-hot-vector encoding of the estimated label and the correlation between the objects in the image. We used an MS-COCO detection dataset to verify the performance of the proposed SSL method and deformable neural networks (D-ConvNets) as an object detector for basic training. The performance of the existing state-of-the-art detectors (DConvNets, YOLO v2, and single shot multi-box detector (SSD)) can be improved by the proposed SSL method without using the additional model parameter or modifying the network architecture.
BibTeX
@inproceedings{choi2018co,
  author = {Min-Kook Choi and Jaehyeong Park and Jihun Jung and Heechul Jung and Jin-Hee Lee and Woong Jae Won and Woo Young Jung and Jincheol Kim and Soon Kwon},
  title = {{Co-occurrence matrix analysis-based semi-supervised training for object detection}},
  booktitle = {ICIP},
  year = {2018},
  doi = {10.1109/icip.2018.8451360},
}
Figure