HalluciDet: Hallucinating RGB Modality for Person Detection Through Privileged Information

Heitor Rapela Medeiros, Fidel A. Guerrero-Peña, Masih Aminbeidokhti, Thomas Dubail, Eric Granger, Marco Pedersoli

TL;DR

This paper introduces HalluciDet, a novel approach for adapting RGB-trained object detectors to infrared (IR) imagery by "hallucinating" an RGB-like representation from IR data. By leveraging a pre-trained RGB detector as "privileged information," HalluciDet guides the image translation process to focus on enhancing objects of interest, significantly improving IR person detection performance without requiring RGB data during inference.

What Problem Does This Paper Address?

Deep learning models, particularly those for computer vision tasks like object detection, often perform poorly when deployed on data from a different modality than they were trained on. This is especially true for cross-modal applications like detecting people in infrared (IR) images using models trained on visible (RGB) images. While image-to-image translation techniques can bridge this gap, traditional methods often focus on reconstructing the entire image, which can introduce irrelevant details and lead to suboptimal detection performance. The core challenge is to adapt an existing, well-performing RGB detector to a new, visually distinct modality (IR) without the need for extensive labeled IR data or retraining the detector from scratch, while ensuring the translated representation is optimized for the specific detection task.

Methods

HalluciDet employs a two-module system: a Hallucination Network and a pre-trained RGB Detector.

  • Hallucination Network: This network, based on a U-Net architecture with attention blocks, takes an IR image as input and generates a "hallucinated" representation. Crucially, this network is trained to produce representations that are easily processed by the RGB detector.
  • Privileged Information & Hallucination Loss: The key innovation is using a pre-trained RGB object detector (with frozen weights) as "privileged information." Instead of a standard image reconstruction loss, HalluciDet uses a hallucination loss that directly minimizes the detection loss of the RGB detector when applied to the hallucinated IR representation. This loss function incorporates standard object detection terms (classification and regression) but is applied to the output of the hallucination network, guiding it to generate representations that are optimal for detection.
  • End-to-End Training: The system is trained end-to-end, where the Hallucination Network's parameters are updated based on the detection performance of the frozen RGB detector on the hallucinated images. This allows the translation process to be directly optimized for the detection task.

Datasets

The paper evaluates HalluciDet on two publicly available IR/RGB datasets:

  • LLVIP: Contains paired IR and RGB images, with a significant number of samples for training and testing.
  • FLIR ADAS: Another dataset with paired IR and RGB images, specifically used for evaluating person detection.

Key Insights and Ideas

  • Task-Specific Image Translation: The most significant insight is that image translation for cross-modal adaptation should be guided by the downstream task (object detection in this case), rather than generic image reconstruction. This task-specific guidance is what HalluciDet achieves by using the detector's loss.
  • Leveraging "Privileged Information": The paper effectively demonstrates how a pre-trained model (the RGB detector) can act as a powerful source of "privileged information" to guide the training of another model (the hallucination network) for a new modality. This avoids the need for extensive target-domain labels.
  • Enhancing Objects of Interest: HalluciDet's hallucinated representations are shown to emphasize objects of interest (people) while suppressing irrelevant background details, making them more amenable to detection by RGB-trained models.
  • Efficiency in Adaptation: By keeping the RGB detector frozen, HalluciDet offers an efficient way to adapt to new modalities, as it only requires training the relatively smaller hallucination network.

Key Results

  • Improved Detection Accuracy: HalluciDet consistently outperforms various image-to-image translation baselines (e.g., FastCUT, CycleGAN) and traditional IR pre-processing techniques on both LLVIP and FLIR ADAS datasets.
  • Significant Gains with Faster R-CNN: The most substantial improvements were observed when using HalluciDet with the Faster R-CNN detector, showing up to a 17% improvement in Average Precision (AP) compared to pixel inversion.
  • Competitive with Fine-Tuning: HalluciDet achieves performance comparable to, and in some cases superior to, standard fine-tuning of RGB detectors on IR data, especially for the Faster R-CNN detector. For instance, on LLVIP, HalluciDet achieved 88.34% AP@50 with Faster R-CNN, compared to 84.94% AP@50 for fine-tuning.
  • Data Efficiency: The paper shows that HalluciDet can achieve comparable performance to fine-tuned models using significantly less training data (e.g., only 30% of the LLVIP dataset was sufficient).
  • Qualitative Improvements: Visualizations show that HalluciDet generates clearer representations that highlight pedestrians, even in challenging low-light or low-heat signature conditions, while reducing background noise.

Limitations and Future Work

  • Detector Dependence: While HalluciDet shows strong results, its performance is influenced by the choice of the pre-trained RGB detector. The paper notes that the improvement is particularly pronounced for two-stage detectors like Faster R-CNN.
  • Hyperparameter Sensitivity: The paper includes an ablation study on the loss weights ($\lambda_{cls}$, $\lambda_{reg}$), indicating that balancing these terms is important for optimal performance.
  • Future Work: The authors suggest exploring different encoder backbones for the hallucination network and investigating the method's effectiveness on other cross-modal detection tasks. They also highlight the potential for HalluciDet to maintain performance in both RGB and IR modalities simultaneously.

Summary updated

Abstract

A powerful way to adapt a visual recognition model to a new domain is through image translation. However, common image translation approaches only focus on generating data from the same distribution as the target domain. Given a cross-modal application, such as pedestrian detection from aerial images, with a considerable shift in data distribution between infrared (IR) to visible (RGB) images, a translation focused on generation might lead to poor performance as the loss focuses on irrelevant details for the task. In this paper, we propose HalluciDet, an IR-RGB image translation model for object detection. Instead of focusing on reconstructing the original image on the IR modality, it seeks to reduce the detection loss of an RGB detector, and therefore avoids the need to access RGB data. This model produces a new image representation that enhances objects of interest in the scene and greatly improves detection performance. We empirically compare our approach against state-of-the-art methods for image translation and for fine-tuning on IR, and show that our HalluciDet improves detection accuracy in most cases by exploiting the privileged information encoded in a pre-trained RGB detector. Code: https://github.com/heitorrapela/HalluciDet.
BibTeX
@inproceedings{medeiros2024hallucidet,
  author = {Heitor Rapela Medeiros and Fidel A. Guerrero-Peña and Masih Aminbeidokhti and Thomas Dubail and Eric Granger and Marco Pedersoli},
  title = {{HalluciDet: Hallucinating RGB Modality for Person Detection Through Privileged Information}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2024},
  doi = {10.1109/wacv57701.2024.00147},
}
Figure