Few-shot concealed object detection in sub-THz security images using improved pseudo-annotations

Ran Cheng, Stepan LucyszynScientific 2024 · DOI

TL;DR

This paper tackles the challenge of detecting concealed objects in sub-terahertz (sub-THz) security images, a domain with scarce labeled data. They introduce an "Improved Pseudo-Annotation" (IPA) method that leverages unlabeled sub-THz images to generate high-quality training data, significantly boosting the performance of few-shot object detection models. This advancement is crucial for real-world security applications where dangerous items are infrequent.

What Problem Does This Paper Address?

Object detection models, particularly deep learning approaches like Convolutional Neural Networks (CNNs), typically require vast amounts of labeled data to achieve high accuracy. However, acquiring and annotating sub-terahertz (sub-THz) images is a costly and time-consuming process. This scarcity is exacerbated in security scenarios where concealed dangerous items (like guns or knives) are inherently rare, leading to class imbalance issues. Traditional "few-shot object detection" (FSOD) methods, designed for limited data, struggle with the unique domain characteristics of sub-THz imaging, particularly the "domain gap" between natural images and sub-THz data. Existing FSOD frameworks, like LVC, which use pseudo-annotations, falter due to this domain gap, leading to degraded performance. This paper aims to bridge this gap by developing a robust pseudo-annotation strategy specifically for sub-THz concealed object detection.

Methods

The core of this work is the Improved Pseudo-Annotation (IPA) method. It builds upon the Label-Verify-Correct (LVC) framework but introduces key modifications to address the sub-THz domain gap.

  • Multiple One-Class Detectors: Instead of a single fine-tuned detector, IPA employs multiple one-class detectors. Each detector is trained on a specific category of concealed object (e.g., guns, knives). This approach helps mitigate class imbalance by allowing the model to focus on individual object types.
  • Fine-Grained Classifier: A fine-grained classifier, pre-trained on thermal-infrared (TIR) images and then fine-tuned on sub-THz data, is used for verification. This classifier is designed to re-verify initial candidates, providing more reliable label assignments, especially for challenging objects with similar appearances. The use of TIR images for pre-training is inspired by their shared spatial features with sub-THz images, acting as a regularization technique to prevent overfitting.
  • Addressing Domain Gap: By using one-class detectors trained on sub-THz data and a classifier pre-trained on related TIR data, IPA aims to reduce the reliance on purely RGB-trained models, thereby narrowing the domain gap that plagued previous methods like LVC.
  • Two-Stage Fine-Tuning (TFA): The paper utilizes the TFA framework as a baseline for training. TFA involves an initial "base training" stage on abundant "base classes" and a subsequent "fine-tuning" stage on limited "novel classes."

Datasets

  • Sub-THz Imaging Dataset: An open-source dataset from the China Academy of Engineering Physics, containing 3157 images captured at 140 GHz with a 335 x 880 pixel resolution. This dataset features human models with concealed objects and includes 11 classes of items.
  • Thermal-Infrared (TIR) Image Dataset: A dataset of 200 thermal-infrared images from Imperial College London, featuring common objects like scissors, toy guns, daggers, and ceramic knives. This dataset was used for pre-training the fine-grained classifier.

Key Insights and Ideas

  • Domain Gap is Critical: The study highlights that directly applying models trained on natural RGB images (like the ViT in LVC) to sub-THz data creates a significant domain gap, hindering performance.
  • Specialized Detectors for Scarcity: Using multiple one-class detectors, each specialized for a specific object category, is more effective than a general detector when dealing with rare "novel classes" in FSOD.
  • TIR Pre-training for Regularization: Pre-training the classifier on thermal-infrared images provides a valuable regularization effect, improving robustness and reducing overfitting, especially when the target sub-THz dataset is small.
  • Pseudo-Annotations are Key: The success of IPA demonstrates that carefully generated pseudo-annotations from unlabeled data can significantly augment limited training sets for challenging domains like sub-THz imaging.

Key Results

The IPA method demonstrated a significant improvement in detecting "novel classes" objects. Specifically, the mean average precision (mAP) for Faster-RCNN improved by almost 30%, from 22.8% to 29.5%, with a small few-shot number (K=10). The paper also shows that IPA outperforms other FSOD strategies (SRC, TFA, LVC) across various few-shot settings, particularly in terms of nAP50 (novel class AP at IoU threshold 50%). For instance, with K=10, IPA achieved an nAP50 of 36.1%, compared to TFA (30.5%), LVC (28.7%), and SRC (25.2%). The precision-recall curves in Figure 7 further illustrate IPA's superior performance across different object categories.

Limitations and Future Work

  • Dataset Dependency: The quality of pseudo-annotations is still dependent on the quality and size of the training dataset.
  • Few-Shot Performance: The IPA method's performance is notably weaker when the few-shot number (K) is very small (e.g., K=1), as there's insufficient data to train a robust classifier.
  • Future Directions: The authors suggest that further enhancements could be achieved by incorporating additional close-support domain images from different sub-THz devices. They also propose exploring advanced deep learning techniques like domain adaptation (DA) to better transfer capabilities from the visual domain to the sub-THz domain.

Summary updated

Abstract

In this research, we explore the few-shot object detection application for identifying concealed objects in sub-terahertz security images, using fine-tuning based frameworks. To adapt these machine learning frameworks for the (sub-)terahertz domain, we propose an innovative pseudo-annotation method to augment the object detector by sourcing high-quality training samples from unlabeled images. This approach employs multiple one-class detectors coupled with a fine-grained classifier, trained on supporting thermal-infrared images, to prevent overfitting. Consequently, our approach enhances the model's ability to detect challenging objects (e.g., 3D-printed guns and ceramic knives) when few-shot training examples are available, especially in the real-world scenario where images of concealed dangerous items are scarce.
BibTeX
@inproceedings{cheng2024few,
  author = {Ran Cheng and Stepan Lucyszyn},
  title = {{Few-shot concealed object detection in sub-THz security images using improved pseudo-annotations}},
  booktitle = {Scientific},
  year = {2024},
  doi = {10.1038/s41598-024-53045-9},
}

Loading related content...

Figure