Multispectral-Polarization-Infrared Multi-Aperture Camouflaged Vehicle Detection

Shu Wang, Xiong Lin, Yuan Song, Junjie Wang, Feng Huang, Wenxi LiuarXiv 2026 · arXiv

TL;DR

This paper introduces a real-time, tri-modal detection framework that combines multispectral, polarization, and infrared imaging to unmask camouflaged vehicles. By shifting from computationally heavy pixel-level alignment to a "consensus-guided" decision-level fusion, the system achieves high accuracy (82.7% mAP) with a blistering 16ms latency, making it ideal for deployment on UAVs.

What Problem Does This Paper Address?

Detecting camouflaged vehicles is a game of cat-and-mouse. Modern camouflage isn't just about matching colors; it uses specialized coatings, nets, and thermal insulation to suppress signatures across the visible and infrared spectrums. A vehicle might be invisible to a standard camera but stand out in its polarization signature (due to man-made surface roughness) or its multispectral reflectance.

While multimodal sensing is the obvious solution, it faces two major hurdles:

  1. Registration Bottlenecks: Most AI models require "pixel-perfect" alignment between different cameras. On a vibrating UAV with different lenses, this is computationally expensive and often fails.
  2. Information Imbalance: Simply "stacking" data from different sensors can lead to noise from one modality drowning out the signal from another.

The authors aim to build a system that is robust to slight misalignments and fast enough for real-time aerial reconnaissance.

Methods

The researchers co-designed a hardware-software solution. On the hardware side, they built a multi-aperture system mounted on a UAV, featuring synchronized snapshot cameras for multispectral, polarization, and thermal infrared data.

Fig. 1. UAV-captured unregistered polarization (visible band), multispectral (NIR), and thermal infrared images of a camouflaged vehicle, where the three modalities have different native resolutions before registration and the vehicle appears to be extremely small from aerial view. The top-left corner shows the magnified region of interest where the vehicle locates.

The software pipeline, dubbed the Consensus-Guided Decision-Level Fusion, works in three main steps:

  1. Optics-Aware Encoding: Instead of feeding raw data into a neural network, they re-encode each modality to highlight specific physical cues. For example, they select the most discriminative bands from multispectral data and use decomposed polarization components (0°, 45°, 90°) to capture material-dependent differences.
  2. Adaptive Bounding-Box Alignment: Rather than aligning every pixel in the images, they only align the detections (bounding boxes). They use offline-calculated homography matrices and an "adaptive box expansion" strategy to account for small registration errors caused by the different viewpoints of the lenses.
  3. Three-Stage Consensus Fusion: This is the "brain" of the system. It doesn't just average the results; it acts like a jury:
  4. Stage 1: Pairs detections from two modalities (e.g., IR and Polarization).
  5. Stage 2: Checks if the third modality (Spectral) supports the pair.
  6. Stage 3: Applies a voting rule. If at least two modalities agree, the target is confirmed. If only one modality sees something, it must have a very high confidence score to be accepted.
The three-stage multimodal target fusion and decision strategy.

Datasets

The authors contributed a new benchmark: the Multimodal Camouflaged Vehicle Detection (CVD) Dataset.

  • Content: 3,726 synchronized samples across five categories (Metal Tank, Plastic Tank, Metal Missile Vehicle, Plastic Missile Vehicle, and Jeep).
  • Challenge: The targets are "tiny," occupying only 0.0139% to 0.0305% of the total image area, captured from an aerial UAV perspective.
  • Generalization: They also tested their framework on the public MID-1K dataset, which uses a different modality mix (RGB-Depth-Infrared) to detect pedestrians indoors.

Key Insights and Ideas

  • Physics over Pixels: The core "aha" moment is that you don't need perfect image alignment if your fusion logic is smart enough at the object level. This bypasses the most computationally expensive part of traditional multimodal pipelines.
  • Modality Complementarity: The paper highlights that different materials fail in different ways. For instance, metal tanks often disappear in thermal IR due to low emissivity but stand out sharply in polarization images. By using three modalities, the system ensures that if one "eye" is blinded by camouflage, the others can still see.
  • Plug-and-Play Architecture: Because the fusion happens at the decision level, the individual detectors (they used YOLOv5s) can be swapped out for newer or different models without redesigning the entire system.

Key Results

  • Accuracy: The proposed method achieved 82.7% [email protected] on the CVD dataset, outperforming state-of-the-art feature-level fusion methods like CTF and DEYOLO.
  • Speed: The end-to-end latency on an NVIDIA Jetson AGX Orin (a common UAV computer) is just 16 ms. This is significantly faster than methods requiring dense pixel registration.
  • Robustness: On the MID-1K dataset, the framework improved mAP by 5.7% over existing methods without any architectural changes, proving it isn't just "overfit" to one specific sensor setup.

Limitations and Future Work

While impressive, the authors note several areas for improvement:

  • Environmental Factors: Performance can still be degraded by extreme motion blur, heavy cloud cover, or fog, which affects all optical sensors.
  • Platform Gap: There is still a performance gap when moving from controlled datasets to high-speed, real-world deployments where vibration and lighting change rapidly.
  • Future Directions: The team plans to extend the framework to more challenging scenes and incorporate lightweight fine-tuning methods to adapt the system to specific platforms more easily.

Summary updated

Abstract

Camouflaged vehicle detection is challenging because camouflage suppresses discriminative cues across sensing modalities. Existing multimodal methods often require precise pixel-level registration and feature-level fusion, creating a trade-off between accuracy and latency. We propose a spectral-polarization-infrared tri-modal detection framework co-designed with a multi-aperture imaging setup to exploit complementary imaging physics. A consensus-guided decision-level fusion strategy combines cross-modal detections within bounded registration tolerances, reducing dependence on precise alignment while maintaining accuracy and real-time performance. We construct a tri-modal camouflaged vehicle dataset containing infrared, polarization, and spectral data. Experiments show that the proposed method achieves 82.7% mean Average Precision (mAP) with an end-to-end latency of 16 ms, outperforming representative feature-level fusion approaches. On the public MID-1K dataset, the framework generalizes to RGB-Depth-Infrared modalities without architectural modification and improves [email protected] by 5.7% over state-of-the-art methods. These results demonstrate the potential of decision-level consensus fusion and multi-aperture sensing for robust and deployable camouflaged vehicle detection.
BibTeX
@misc{wang2026multispectral,
  author = {Shu Wang and Xiong Lin and Yuan Song and Junjie Wang and Feng Huang and Wenxi Liu},
  title = {{Multispectral-Polarization-Infrared Multi-Aperture Camouflaged Vehicle Detection}},
  year = {2026},
  eprint = {2611.22246},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.22246},
}
Figure