MAVNet: an Effective Semantic Segmentation Micro-Network for MAV-based Tasks

Ty Nguyen, Tolga Özaslan, Ian D. Miller, James Keller, Shreyas S. Shivakumar, Giuseppe Loianno, Camillo J. Taylor, Vijay Kumar, Joseph H. Harwood, Jennifer M. Wozencraft

Abstract

Real-time semantic image segmentation on platforms subject to size, weight, and power constraints is a key area of interest for air surveillance and inspection. In this letter, we propose MAVNet: a small, light-weight, deep neural network for real-time semantic segmentation on micro aerial vehicles (MAVs). MAVNet, inspired by ERFNet, features 400 times fewer parameters and achieves comparable performance with some reference models in empirical experiments. Additionally, we provide two novel datasets that represent challenges in semantic segmentation for real-time MAV tracking and infrastructure inspection tasks and verify MAVNet on these datasets. Our algorithm and datasets are made publicly available.

Introduction

Semantic segmentation—the process of assigning a class label to every pixel in an image—is a critical capability for autonomous Micro Aerial Vehicles (MAVs). It enables tasks such as target tracking, obstacle avoidance, and infrastructure inspection. However, MAVs operate under strict SWaP constraints (Size, Weight, and Power), which often preclude the use of state-of-the-art models like ResNet or UNet that require significant GPU memory and power.

MAV platforms used for evaluation in laboratory (left) and penstock (right) environments.

Furthermore, standard benchmarks like CityScapes do not account for the unique challenges of aerial perspectives, such as motion blur and low-light conditions in confined spaces like hydroelectric dam penstocks. MAVNet addresses these issues by providing a micro-network optimized for onboard embedded hardware.

MAVNet Architecture

MAVNet is designed for extreme efficiency without sacrificing the necessary receptive field for segmentation. Key architectural innovations include:

  • Depth-wise Feature Aggregation Block (DWFab): The core building block of MAVNet, combining depth-wise separable convolutions to reduce computation with dilated convolutions to expand the network's view without adding layers.
  • Shallow Decoder: To minimize latency, MAVNet uses a truncated decoder with nearest-neighbor upsampling, avoiding the heavy computational cost of traditional deconvolutional layers.
  • Aggressive Downsampling: The network reduces resolution early in the pipeline to minimize the workload for subsequent layers.
  • Focal Loss Training: This specialized loss function helps the network focus on small, hard-to-detect objects (like a distant drone) rather than being overwhelmed by easy-to-classify background pixels.

Datasets and Benchmarking

The researchers introduced two new datasets to evaluate MAV-specific performance:

  1. MAV Segmentation Dataset: Focused on indoor multi-robot tracking and formation control.
  2. Penstock Dataset: Captured inside the Center Hill Dam, requiring the identification of rivets, corrosion, and water in dark, metallic tunnels.
Example from the MAV tracking dataset showing the source image, zoom-ins on targets, and the ground truth segmentation mask.

Key Results

MAVNet demonstrates that "less is more" for specialized aerial tasks. With only ~4,300 parameters—400x fewer than ERFNet—it achieves real-time performance on embedded systems:

  • Speed: 9 FPS on NVIDIA Jetson Xavier and 4.4 FPS on the older Jetson TX2.
  • Accuracy: On the Penstock dataset, MAVNet achieved the highest average Intersection over Union (IoU) across all classes compared to larger models.
  • Tracking Precision: For MAV tracking, MAVNet excelled in Centroid Distance accuracy. This metric is vital for flight controllers, as identifying the center of a target is more important for following a lead drone than perfect boundary segmentation.

Conclusion and Future Work

MAVNet proves that micro-networks can effectively handle complex semantic segmentation tasks on SWaP-constrained platforms. While the current architecture has a higher false-positive rate due to the lack of long skip connections, it provides a viable path for fully autonomous onboard intelligence. Future research will focus on refining the network to reduce noise and optimizing the implementation for specialized hardware tools like NVIDIA TensorRT.

Summary updated

Abstract

Real-time semantic image segmentation on platforms subject to size, weight and power (SWaP) constraints is a key area of interest for air surveillance and inspection. In this work, we propose MAVNet: a small, light-weight, deep neural network for real-time semantic segmentation on micro Aerial Vehicles (MAVs). MAVNet, inspired by ERFNet, features 400 times fewer parameters and achieves comparable performance with some reference models in empirical experiments. Our model achieves a trade-off between speed and accuracy, achieving up to 48 FPS on an NVIDIA 1080Ti and 9 FPS on the NVIDIA Jetson Xavier when processing high resolution imagery. Additionally, we provide two novel datasets that represent challenges in semantic segmentation for real-time MAV tracking and infrastructure inspection tasks and verify MAVNet on these datasets. Our algorithm and datasets are made publicly available.
BibTeX
@article{nguyen2019mavnet,
  author = {Ty Nguyen and Tolga Özaslan and Ian D. Miller and James Keller and Shreyas S. Shivakumar and Giuseppe Loianno and Camillo J. Taylor and Vijay Kumar and Joseph H. Harwood and Jennifer M. Wozencraft},
  title = {{MAVNet: an Effective Semantic Segmentation Micro-Network for MAV-based Tasks}},
  journal = {IEEE Robotics Autom. Lett.},
  year = {2019},
  url = {https://openreview.net/forum?id=wcTchkke2IP},
}
Figure