Abstract
Real-time semantic image segmentation on platforms subject to size, weight, and power constraints is a key area of interest for air surveillance and inspection. In this letter, we propose MAVNet: a small, light-weight, deep neural network for real-time semantic segmentation on micro aerial vehicles (MAVs). MAVNet, inspired by ERFNet, features 400 times fewer parameters and achieves comparable performance with some reference models in empirical experiments. Additionally, we provide two novel datasets that represent challenges in semantic segmentation for real-time MAV tracking and infrastructure inspection tasks and verify MAVNet on these datasets. Our algorithm and datasets are made publicly available.
Introduction
Semantic segmentation—the process of assigning a class label to every pixel in an image—is a critical capability for autonomous Micro Aerial Vehicles (MAVs). It enables tasks such as target tracking, obstacle avoidance, and infrastructure inspection. However, MAVs operate under strict SWaP constraints (Size, Weight, and Power), which often preclude the use of state-of-the-art models like ResNet or UNet that require significant GPU memory and power.
Furthermore, standard benchmarks like CityScapes do not account for the unique challenges of aerial perspectives, such as motion blur and low-light conditions in confined spaces like hydroelectric dam penstocks. MAVNet addresses these issues by providing a micro-network optimized for onboard embedded hardware.
MAVNet Architecture
MAVNet is designed for extreme efficiency without sacrificing the necessary receptive field for segmentation. Key architectural innovations include:
- Depth-wise Feature Aggregation Block (DWFab): The core building block of MAVNet, combining depth-wise separable convolutions to reduce computation with dilated convolutions to expand the network's view without adding layers.
- Shallow Decoder: To minimize latency, MAVNet uses a truncated decoder with nearest-neighbor upsampling, avoiding the heavy computational cost of traditional deconvolutional layers.
- Aggressive Downsampling: The network reduces resolution early in the pipeline to minimize the workload for subsequent layers.
- Focal Loss Training: This specialized loss function helps the network focus on small, hard-to-detect objects (like a distant drone) rather than being overwhelmed by easy-to-classify background pixels.
Datasets and Benchmarking
The researchers introduced two new datasets to evaluate MAV-specific performance:
- MAV Segmentation Dataset: Focused on indoor multi-robot tracking and formation control.
- Penstock Dataset: Captured inside the Center Hill Dam, requiring the identification of rivets, corrosion, and water in dark, metallic tunnels.
Key Results
MAVNet demonstrates that "less is more" for specialized aerial tasks. With only ~4,300 parameters—400x fewer than ERFNet—it achieves real-time performance on embedded systems:
- Speed: 9 FPS on NVIDIA Jetson Xavier and 4.4 FPS on the older Jetson TX2.
- Accuracy: On the Penstock dataset, MAVNet achieved the highest average Intersection over Union (IoU) across all classes compared to larger models.
- Tracking Precision: For MAV tracking, MAVNet excelled in Centroid Distance accuracy. This metric is vital for flight controllers, as identifying the center of a target is more important for following a lead drone than perfect boundary segmentation.
Conclusion and Future Work
MAVNet proves that micro-networks can effectively handle complex semantic segmentation tasks on SWaP-constrained platforms. While the current architecture has a higher false-positive rate due to the lack of long skip connections, it provides a viable path for fully autonomous onboard intelligence. Future research will focus on refining the network to reduce noise and optimizing the implementation for specialized hardware tools like NVIDIA TensorRT.
Summary updated