TL;DR
This paper introduces MonoSeg, a novel instance segmentation model designed for infrared images captured by drones, specifically for detecting vehicles. It tackles the challenges of low contrast, varying viewpoints, and computational constraints by introducing new modules for feature extraction and integration, along with an improved loss function, achieving state-of-the-art accuracy with significantly reduced computational cost.
What Problem Does This Paper Address?
Unmanned Aerial Vehicles (UAVs) equipped with infrared cameras are increasingly used for surveillance and monitoring, especially in low-light conditions. However, accurately identifying and segmenting vehicles in infrared imagery from a UAV perspective is difficult due to several factors:
- Infrared Imaging Limitations: Low resolution, weak contrast between vehicles and backgrounds, and thermal crossover effects make it hard to distinguish targets.
- Dynamic UAV Perspective: Varying altitudes, angles, and imaging times introduce significant viewpoint changes.
- Computational Constraints: UAVs have limited onboard processing power, requiring models that are both accurate and computationally efficient for real-time operation.
- Lack of Suitable Datasets: Existing datasets for UAV-based infrared vehicle instance segmentation are scarce.
While object detection is common, instance segmentation (which provides pixel-level masks for each object) offers more detailed information but is less adopted in this domain due to the aforementioned challenges.
Methods
The authors propose MonoSeg, an improved instance segmentation framework built upon the YOLOv8n-Seg baseline. Their approach focuses on three key areas:
- Ghost Feature Bottle Cross (GFBC) Module: This module is integrated into the backbone of the network. It uses "Ghost Convolutions" to generate more semantic features with fewer computations, reducing the computational burden and improving feature extraction efficiency. It also incorporates a "Bottle Cross" structure to create a multi-level receptive field hierarchy, enhancing the model's ability to perceive targets at different scales.
- Scale Feature Recombination (SFR) Module: This module is placed in the "neck" of the network, responsible for integrating features from different scales. It uses downsampling, upsampling, and channel shuffling to recombine high-density tensor information, improving feature utilization and selection without significant computational overhead.
- Comprehensive Loss Function (C-Loss): To address issues with imbalanced positive and negative samples and slow convergence in mask prediction, they propose a combined loss function. This includes a "Mask Weighted BCE Loss" to better balance foreground and background contributions and a "Mask Dice Loss" to improve the convergence speed and accuracy of mask segmentation, especially for positive pixels.
Datasets
The authors introduce a new dataset called DIVIS (Drone Infrared Vehicle Instance Segmentation dataset). This dataset is constructed from four sources:
- Aerial-Mancar [37]
- VisDrone [38]
- HIT-UAV [39]
- Self-collected infrared images from their own UAV.
The DIVIS dataset contains 3303 single-channel infrared images with a resolution of 640x512. It features two categories of vehicle targets: "small_vehicle" (sedans, vans, trucks) and "large_vehicle" (large trucks, buses, trailers). The dataset aims to cover diverse viewpoints, altitudes, and wavelength types, presenting challenges due to significant differences in target size and aspect ratio.
Key Insights and Ideas
- Balancing Act: The core challenge is achieving high accuracy in infrared vehicle segmentation while keeping computational costs low for UAV deployment. MonoSeg demonstrates that this balance is achievable.
- Ghost Convolutions for Efficiency: The GFBC module effectively leverages Ghost Convolutions to reduce computation without sacrificing feature representation quality, a crucial aspect for resource-constrained platforms.
- Multi-Scale Fusion is Key: The SFR module highlights the importance of intelligently fusing features from different scales in the neck of the network to improve information utilization.
- Loss Function Matters for Masks: The C-Loss function shows that carefully designed loss terms can significantly improve the precision and robustness of pixel-level mask predictions, especially for challenging scenarios.
- Dataset Diversity is Crucial: The creation of the DIVIS dataset addresses a critical gap, providing a benchmark specifically tailored for UAV infrared vehicle instance segmentation, encompassing varied conditions.
Key Results
- Accuracy: MonoSeg achieves state-of-the-art performance on the DIVIS dataset, outperforming several baseline models. For instance, it shows a 2.4% improvement in mAP Box0.5 and a 2.5% improvement in mAP Mask0.5 compared to the YOLOv8n-Seg baseline.
- Speed: The model maintains a high inference speed, achieving 110 FPS, which is competitive with other lightweight models and significantly faster than some high-accuracy but computationally intensive methods.
- Computational Cost: MonoSeg reduces computational cost by 0.4 GFLOPs compared to the baseline, making it suitable for embedded systems.
- Adaptability: The model demonstrates a well-balanced performance across different object sizes (small vs. large vehicles) and under various IoU thresholds, indicating strong adaptability.
- Qualitative Improvements: Visualization experiments show MonoSeg provides more complete and balanced recognition of multiple targets, with better foreground-background discrimination and fewer missed detections compared to other methods.
Limitations and Future Work
- Fine-grained Pixel-level Segmentation: The authors acknowledge that MonoSeg, like its YOLO-based predecessors, faces challenges in achieving extremely fine-grained pixel-level segmentation. Future work aims to integrate low-level pixel semantics with high-level prototypes for more refined segmentation.
- Fixed Anchor Density: The reliance on a fixed number of anchors can limit effectiveness in scenarios with extremely dense targets. Future research will focus on adaptively adjusting anchor density.
- Dataset Expansion: The authors plan to expand datasets to cover a broader spectrum of scenarios and objects.
- Algorithm Optimization: Further optimization of the algorithm design is planned to enhance precision and speed.
- Deployment on Embedded Systems: The ultimate goal is to deploy and optimize these algorithms on embedded systems within UAVs for real-world edge applications.
Summary updated