Co-Occurrence Matrix Analysis-Based Semi-Supervised Training for Object Detection

Min-Kook Choi, Jaehyeong Park, Jihun Jung, Heechul Jung, Jinhee Lee, Woong Jae Won, Woo Young Jung, Jin-Cheol Kim, Soon Kwon

Abstract

One of the most important factors in training object recognition networks using convolutional neural networks (CNN) is the provision of annotated data accompanying human judgment. Particularly, in object detection or semantic segmentation, the annotation process requires considerable human effort. In this paper, we propose a semi-supervised learning (SSL)-based training methodology for object detection, which makes use of automatic labeling of un-annotated data by applying a network previously trained from an annotated dataset. Because an inferred label by the trained network is dependent on the learned parameters, it is often meaningless for re-training the network. To transfer a valuable inferred label to the unlabeled data, we propose a re-alignment method based on co-occurrence matrix analysis that takes into account one-hot-vector encoding of the estimated label and the correlation between the objects in the image. We used an MS-COCO detection dataset to verify the performance of the proposed SSL method and deformable neural networks (D-ConvNets) as an object detector for basic training. The performance of the existing state-of-the-art detectors (D-ConvNets, YOLO v2, and SSD) can be improved by the proposed SSL method without using the additional model parameter or modifying the network architecture.

Introduction

Deep learning models for object detection require vast amounts of annotated data, which is labor-intensive to produce. Semi-supervised learning (SSL) aims to mitigate this by using a small labeled dataset to generate "pseudo-labels" for a larger unlabeled set. However, naive pseudo-labeling often suffers from error propagation, where the model reinforces its own mistakes—a phenomenon known as the "garbage in, garbage out" loop. This paper addresses this challenge by introducing a refinement method using co-occurrence matrix analysis to validate labels based on contextual relationships between objects.

Proposed Methodology

The proposed framework operates in two distinct phases to ensure high-quality label generation. In Phase 1, a base detector (such as Deformable ConvNets) is trained on a standard annotated dataset. This model then performs inference on an unlabeled dataset to generate initial pseudo-labels. In Phase 2, these labels are refined through statistical analysis and used to retrain the model from scratch, effectively expanding the training set with reliable synthetic data.

The two-phase semi-supervised learning pipeline including initial training, pseudo-labeling, and retraining.

Co-Occurrence Matrix Analysis

The core innovation of this work is the refinement of pseudo-labels using a co-occurrence matrix. This matrix, derived from the initial training data, captures the statistical likelihood of different object classes appearing together (e.g., a "keyboard" frequently co-occurs with a "mouse"). By applying conditional marginalization and max-normalization, the system can filter out statistically improbable detections and reinforce those that align with known contextual patterns. This acts as a rudimentary form of "common sense" for the detector, preventing it from accepting high-confidence but contextually impossible errors.

Refining pseudo-labels using co-occurrence matrix analysis, conditional marginalization, and max-normalization.

Experimental Setup and Results

The method was evaluated using the MS-COCO dataset, specifically utilizing COCO2017 (train+val) for initial training and the unlabeled2017 set (123k images) for the SSL process. Final evaluation was performed on the test-dev17 benchmark. The proposed SSL method consistently improved mean Average Precision (mAP) across various architectures without requiring architectural modifications:

  • D-ConvNets: Improved from 36.3 to 37.8 mAP (+1.5).
  • YOLO v2: Improved from 24.0 to 25.1 mAP (+1.1).
  • SSD: Improved from 24.1 to 25.3 mAP (+1.2).

Conclusion

This research presents a model-agnostic SSL framework that leverages contextual relationships via co-occurrence matrices. By effectively cleaning pseudo-labels, the method improves the performance of state-of-the-art detectors. While the approach increases training time due to the larger dataset size, it offers a significant performance boost without the need for additional manual annotation. Future work will focus on optimizing hyperparameter sensitivity and exploring the interaction between this SSL scheme and different detector architectures.

Summary updated

Abstract

One of the most important factors in training object recognition networks using convolutional neural networks (CNN) is the provision of annotated data accompanying human judgment. Particularly, in object detection or semantic segmentation, the annotation process requires considerable human effort. In this paper, we propose a semi-supervised learning (SSL)-based training methodology for object detection, which makes use of automatic labeling of un-annotated data by applying a network previously trained from an annotated dataset. Because an inferred label by the trained network is dependent on the learned parameters, it is often meaningless for re-training the network. To transfer a valuable inferred label to the unlabeled data, we propose a re-alignment method based on co-occurrence matrix analysis that takes into account one-hot-vector encoding of the estimated label and the correlation between the objects in the image. We used an MS-COCO detection dataset to verify the performance of the proposed SSL method and deformable neural networks (D-ConvNets) [1] as an object detector for basic training. The performance of the existing state-of-the-art detectors (D-ConvNets, YOLO v2 [2], and single shot multi-box detector (SSD) [3]) can be improved by the proposed SSL method without using the additional model parameter or modifying the network architecture.
BibTeX
@inproceedings{choi2018co,
  author = {Min-Kook Choi and Jaehyeong Park and Jihun Jung and Heechul Jung and Jinhee Lee and Woong Jae Won and Woo Young Jung and Jin-Cheol Kim and Soon Kwon},
  title = {{Co-Occurrence Matrix Analysis-Based Semi-Supervised Training for Object Detection}},
  booktitle = {ICIP},
  year = {2018},
  doi = {10.1109/icip.2018.8451360},
}
Figure