TL;DR
This paper introduces YOLO Phantom, a significantly smaller and faster YOLO model designed for resource-constrained IoT devices. By introducing a novel "Phantom Convolution" block and optimizing the YOLOv8 architecture, YOLO Phantom achieves comparable accuracy to larger models while drastically reducing computational demands, making real-time multimodal object detection feasible on edge devices.
What Problem Does This Paper Address?
Object detection in real-world IoT applications, such as autonomous vehicles and security systems, is severely hampered by low-light conditions and occlusions. While advanced deep learning models offer high accuracy, their substantial computational requirements make them unsuitable for resource-limited edge devices, preventing real-time deployment. Existing solutions often struggle to balance accuracy with efficiency, especially when dealing with multimodal data (like RGB and thermal imagery) which is crucial for overcoming these challenges.
Methods
The core of this work is the development of YOLO Phantom, a modified version of the Ultralytics YOLOv8n architecture. The key innovations include:
- Phantom Convolution Block: This novel block is designed to be more efficient than existing Ghost Convolution blocks. It achieves this by combining a modified depthwise separable convolution with a modified group convolution. This approach reduces parameters and computational cost while aiming to maintain or improve feature extraction capabilities.
- Architectural Modifications: The YOLOv8 backbone and neck were optimized. Specifically, the C2f blocks were modified to "C2fi" blocks by removing shortcut connections and reducing the number of filters in deeper layers. The seventh layer of the backbone and the final convolution block in the head were replaced with the new Phantom Convolution block.
- Multimodal Fusion: The model is designed to handle both RGB and thermal imagery. It was trained using a multimodal dataset, enabling it to leverage the complementary information from different sensor types.
- End-to-End System: The researchers built a practical system integrating a Raspberry Pi with RGB and NoIR cameras, connected to an AWS IoT Core for real-time notifications upon object detection.
Datasets
- FLIR V2 Dataset: This dataset, containing both RGB and thermal images, was used as the primary source for creating a multimodal training dataset.
- Custom Multimodal Dataset: The authors curated a training set of 10,478 thermal and 10,053 RGB images, and a validation set of 1,128 thermal and 1,069 RGB images from FLIR V2. They note that the original FLIR V2 dataset's RGB and thermal images are not always from the exact same scene, so their curated dataset mixes scenes to help the model generalize and avoid misalignment.
- Test Datasets: Separate thermal (3,749 images) and RGB (3,493 images) test datasets were constructed from the FLIR V2 test section.
Key Insights and Ideas
- Phantom Convolution is a Game Changer for Efficiency: The novel Phantom Convolution block is the cornerstone of YOLO Phantom's success. By cleverly combining depthwise separable and group convolutions, it achieves significant computational savings without sacrificing accuracy, outperforming previous efficient convolution designs.
- Multimodal Fusion is Crucial for Robustness: The ability to fuse RGB and thermal data is essential for handling challenging conditions like low light and occlusion. YOLO Phantom demonstrates that this fusion can be achieved effectively even with a highly optimized, lightweight architecture.
- IoT Edge Deployment is Now More Realistic: The paper effectively bridges the gap between high-accuracy deep learning models and the limitations of edge devices. YOLO Phantom's small size and speed make it a viable solution for real-time object detection in environments where computational power is scarce.
- Transfer Learning on Curated Data Pays Off: Leveraging pre-trained weights and then fine-tuning on a carefully constructed multimodal dataset was key to achieving strong performance across different modalities.
Key Results
- Size and Parameter Reduction: YOLO Phantom achieved a 43% reduction in parameters and model size compared to the YOLOv8n model.
- Computational Efficiency: The model boasts a 19% reduction in GFLOPs.
- Speed Improvements: YOLO Phantom demonstrated a 17% increase in FPS for thermal detection and a 14% increase for RGB detection compared to the baseline YOLOv8n model.
- Accuracy: The model achieved comparable accuracy (mAP) to the latest YOLOv8n model, and in some cases, even surpassed smaller YOLO variants like YOLOv4-tiny, YOLOv7-tiny, and YOLOv5nu, despite having significantly fewer parameters (4-2x fewer).
- Cross-Modality Performance: YOLO Phantom showed robust performance across different modalities, often outperforming unimodal-trained models when tested on the "reverse" modality.
Limitations and Future Work
- Dataset Alignment: The authors acknowledge that their multimodal dataset mixes scenes from the original FLIR V2 dataset, meaning RGB and thermal images are not always perfectly aligned. While this helps generalization, it's a deviation from perfectly synchronized data.
- Future Work: The authors suggest that their advancement holds promise for broader performance gains across computer vision tasks like classification and segmentation, and can pave the way for applications in environmental monitoring and wildlife tracking. They also integrated temperature monitoring and audio alerts for enhanced situational awareness in their end-to-end system.
Summary updated