Abstract
This paper introduces a versatile pedestrian detector designed to operate seamlessly across visible (RGB) and thermal sensor types. By leveraging a novel Multispectral Recalling (MSR) Memory and a specialized contrastive loss, the model can "recall" rich multispectral information even when only a single sensor input is available at inference time. This approach significantly outperforms existing single-modality and multispectral detectors in real-world scenarios where sensor availability may fluctuate.
Introduction
Modern surveillance systems often switch between visible light sensors during the day and thermal sensors at night or in adverse weather. Current computer vision models typically fall into two categories: single-modal specialists that fail when the environment changes, or multispectral detectors that require perfectly aligned image pairs at runtime—a requirement that is computationally expensive and hardware-dependent. This research addresses the challenge of building a modality-agnostic detector capable of high-quality performance using either RGB or thermal inputs without prior knowledge of the input type or paired images at inference.
Methodology
The proposed framework utilizes paired multispectral data during training to "teach" the model, while allowing it to function as a single-modality detector during deployment. The architecture consists of three primary components:
- Multisensory-Matching Contrastive Loss ($L_{mmc}$): This loss forces the backbone to extract modality-invariant features for pedestrians, pushing features from different sensors closer in the embedding space while repelling background noise.
- Multispectral Recalling (MSR) Memory: A key-value memory module that stores compressed "multispectral contexts" learned during training.
- Multispectral Recalling Loss ($L_{msr}$): This guides the memory module to "hallucinate" missing multispectral information during single-modality inference, refining the available features with stored cross-modal knowledge.
Experimental Results
The model was evaluated on the KAIST Multispectral Pedestrian Dataset and the CVC-14 benchmark. Performance is measured using the Miss Rate (MR), where lower values indicate better performance.
The proposed method demonstrates significant robustness in "mixed modality" scenarios, simulating a camera switching from day to night. As shown in the results table, the model achieves a Miss Rate of 15.87% in mixed conditions, substantially outperforming the baseline and other thermal-based detectors. Furthermore, when only a single modality is provided at inference, the method achieves an MR of 11.39%, whereas many state-of-the-art multispectral models fail or are inapplicable.
Discussion and Key Insights
The core innovation lies in using Memory as a Bridge. Instead of forcing one modality to mimic another, the model associates features from both with a shared multispectral prototype. This asymmetry between training (paired data) and inference (single modality) allows the model to act as a "student" that has internalized the "teacher's" (multispectral) knowledge.
Despite its strengths, the model requires well-aligned data during the training phase to avoid storing noisy features in the memory module. Future work may explore scaling the memory slots and applying this "recalling" mechanism to broader multi-modal tasks like general object detection or autonomous driving sensor fusion.
Conclusion
By decoupling the requirement for multispectral inputs at inference from the benefits of multispectral training, this work provides a practical path toward efficient, high-performance pedestrian detection in dynamic real-world environments. The model maintains a computational profile similar to single-modal detectors while delivering the accuracy of complex multispectral systems.
Summary updated