Gram-SLD: Automatic Self-labeling and Detection for Instance Objects

Rui Wang, Chengtun Wu, Jiawen Xin, Liang ZhangarXiv 2021 · arXiv

Abstract

Instance-level object detection is a critical task in computer vision, yet it is often hindered by the high cost of manual data annotation. This paper presents Gram-SLD (Gram Self-Labeling and Detection), a semi-supervised framework designed to achieve near-fully-supervised performance using only a small fraction of labeled data. By integrating a novel Gram loss to ensure detector diversity and an entropy-based sample selection strategy, Gram-SLD effectively leverages unlabeled datasets to generate high-quality training signals. Experimental results across multiple datasets demonstrate that the framework can reach performance levels within 1-2% of fully supervised baselines using only 5% of the manual labels.

Introduction

Deep Convolutional Neural Networks (DCNNs) have set the standard for object detection, but their success is largely predicated on the availability of massive, labeled datasets. While category-level detection (e.g., identifying a generic "can") is well-supported by public datasets like COCO, instance-level detection (e.g., identifying a specific brand of soda) requires specialized labeling for every new environment or object set. This "manual annotation bottleneck" limits the scalability of instance detection in real-world applications like robotics and inventory management.

To address this, researchers have turned to semi-supervised learning, specifically co-training. In a co-training setup, two detectors are trained to teach one another. However, the efficacy of co-training relies on the "sufficient and redundant" assumption: the two models must provide independent views of the data to avoid reinforcing each other's errors. Gram-SLD introduces a mathematical approach to enforce this independence without requiring multiple physical sensors.

Methodology

Gram-SLD consists of three primary innovations: a diversity-enforcing loss function, an intelligent sample selection strategy, and a robust self-labeling mechanism.

Gram Loss for Detector Independence

The core innovation of Gram-SLD is the use of Gram matrices to maintain diversity between two detectors, $D_1$ and $D_2$. In the context of style transfer, Gram matrices are used to capture the "style" or texture of an image. Gram-SLD adapts this concept by defining a "Gram loss" based on the reciprocal of the Mean Square Error (MSE) between the feature map Gram matrices of the two detectors. By minimizing the total loss, the network is forced to maximize the difference between these matrices. This ensures that the two detectors develop independent feature representations—effectively different "views" of the same input—which is essential for successful co-training.

Key Sample Selection

Rather than selecting the initial 5% of data for manual labeling at random, Gram-SLD employs an unsupervised pre-processing step to maximize information density. The framework clusters unlabeled images using HSV color histograms and then selects the "top M" images with the highest entropy from each cluster. This strategy ensures that the initial labeled set is highly representative of the overall dataset distribution, providing a strong foundation for the subsequent self-labeling phases.

Automatic Self-Labeling

During the iterative training process, the two detectors process the unlabeled data pool. A pseudo-label is generated only when both detectors demonstrate high confidence and high Intersection over Union (IoU) agreement. This strategy balances precision and recall, ensuring the model does not ignore difficult objects or mistakenly categorize them as background, which would otherwise lead to performance degradation.

Experimental Evaluation

The framework was evaluated on three distinct datasets to test its robustness across different environments:

  1. GMU Kitchen Dataset: Contains RGB videos of kitchen scenes with 11 specific objects.
  2. Active Vision Dataset (AVD): A large-scale indoor dataset characterized by a high volume of small objects, presenting a significant challenge for detection.
  3. BHID-ITEM Dataset: A custom dataset created by the authors to simulate real-world office environments.

Results

The results indicate that Gram-SLD effectively closes the gap between semi-supervised and fully supervised learning. On the GMU Kitchen Dataset, Gram-SLD achieved 83.8% mAP using only 5% labels, compared to 84.7% mAP for the fully supervised model (100% labels). Similarly, on the BHID-ITEM dataset, the framework reached 83.6% mAP, trailing the fully supervised baseline by only 1.9%.

Notably, the authors found that increasing the labeled data from 5% to 10% yielded diminishing returns. This suggests that the entropy-based selection strategy successfully captures the essential features of the dataset early in the process.

Discussion and Limitations

While Gram-SLD shows remarkable efficiency, it is not without limitations. The performance gap was most pronounced on the AVD dataset (up to 4.4%), which the authors attribute to the difficulty of detecting small objects. Small objects often lead to lower-quality pseudo-labels, which can introduce noise into the training cycle.

Furthermore, the framework relies on an "initial proficiency" threshold. If the starting model accuracy is too low (e.g., below 70% mAP on standard benchmarks), the co-training process risks becoming an "echo chamber" where the models reinforce hallucinations rather than learning true features. Finally, the iterative nature of pseudo-labeling introduces computational overhead compared to standard supervised training.

Conclusion

Gram-SLD provides a practical solution to the data scarcity problem in instance object detection. By leveraging Gram matrices to enforce model diversity and using entropy-based selection to optimize the initial label set, the framework achieves high accuracy with minimal human intervention. Future work will focus on improving the detection of small objects and optimizing the computational efficiency of the iterative training cycles.

Summary updated

BibTeX
@misc{wang2021gram,
  author = {Rui Wang and Chengtun Wu and Jiawen Xin and Liang Zhang},
  title = {{Gram-SLD: Automatic Self-labeling and Detection for Instance Objects}},
  year = {2021},
  eprint = {2112.03641},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2112.03641},
}
Figure