M-FasterSeg: An Efficient Semantic Segmentation Network Based on Neural Architecture Search

Junjun Wu, Huiyu Kuang, Qinghua Lu, Zeqin Lin, Qingwu Shi, Xilin Liu, Xiaoman Zhu

TL;DR

M-FasterSeg is a high-performance semantic segmentation network designed for real-time applications like mobile robots and autonomous driving. By combining Neural Architecture Search (NAS) with a novel adaptive spatial attention module and a teacher-student distillation framework, the authors achieved a significant balance between speed and accuracy, reaching 69.8% mIoU on the Cityscapes dataset at a blistering 166.4 FPS.

What Problem Does This Paper Address?

Semantic segmentation—the task of assigning a category label to every single pixel in an image—is computationally expensive. While state-of-the-art models like the DeepLab series provide incredible accuracy, they are often too slow for the "edge" (drones, mobile robots, or cars) where decisions must be made in milliseconds.

The authors identify three main hurdles in current research:

  1. Manual Design Bottlenecks: Manually designing network architectures is time-consuming and requires expert intuition.
  2. Information Loss: Fast networks often struggle with "edge information," losing the crisp boundaries between objects and backgrounds.
  3. NAS Instability: Automated search methods (NAS) often suffer from "architecture collapse," where the search process converges on poor-quality structures because it over-prioritizes speed or falls into local minima.

Methods

The authors introduce M-FasterSeg, which builds upon the FasterSeg framework but adds several sophisticated layers of optimization:

  • Multi-Resolution NAS: The system uses NAS to automatically find the best network structure across multiple resolution branches. It employs a "space expansion" method, allowing the search to explore not just the topology, but also the width and convolution units of the network.
  • Adaptive Spatial Attention (IRNN): To solve the edge-loss problem, the authors integrated an attention module based on Improved Recurrent Neural Networks (IRNN). This module "sweeps" across the feature map in four directions (up, down, left, right) to capture global context and spatial relationships without the massive overhead of standard Transformers.
  • Teacher-Student Distillation: The NAS process simultaneously searches for two models: a high-complexity "Teacher" and a lightweight "Student." During training, the Teacher transfers its "knowledge" (feature weights) to the Student, allowing the smaller model to perform far better than if it were trained in isolation.
  • Regularized Delay Optimization: To prevent the NAS from collapsing into uselessly simple structures, the authors introduced a latency-aware optimization function. This balances the influence of different operators (convolutions, down-sampling rates, etc.) to ensure the search stays stable.

Datasets

The primary benchmark used is the Cityscapes dataset. This is a standard, high-quality dataset for urban scene understanding, featuring:

  • 5,000 fine-annotated images of street scenes from 50 different cities.
  • Categories including cars, pedestrians, sidewalks, and buildings.
  • A focus on complex, real-world outdoor environments.

Key Insights and Ideas

  • Recurrence for Global Context: One of the "aha" moments is using IRNNs for attention. While most modern networks use heavy self-attention or large kernels to see the "big picture," M-FasterSeg uses simple recurrent passes. This provides a large receptive field very efficiently.
  • Simultaneous Search: Instead of searching for a model and then distilling it later, searching for the Teacher and Student at the same time within the same supernet significantly reduces the total search time and ensures the architectures are compatible for knowledge transfer.
  • Stabilizing the Search: The authors realized that architecture collapse in NAS is often due to the search being too sensitive to specific operators. Their "regularized delay" method acts as a stabilizer, ensuring the computer doesn't "cheat" by picking a fast but inaccurate model.

Key Results

M-FasterSeg demonstrates a superior trade-off between accuracy and efficiency compared to classic and modern baselines:

  • Accuracy: Achieved 69.8% mIoU (mean Intersection over Union) on the Cityscapes test set.
  • Speed: Reached an inference speed of 166.4 FPS on an NVIDIA GTX 1080ti.
  • Efficiency: The "More FasterSeg" variant reduced parameters to 2.90M (down from 3.09M in the non-attention version) while actually increasing accuracy by 1.3%.
  • Qualitative: Visual results show much sharper edge segmentation in complex scenes, such as distinguishing thin poles or distant pedestrians against a busy background.

Limitations and Future Work

While the results are impressive, there are a few areas for further exploration:

  • Hardware Specificity: The latency optimization is often tied to the specific hardware used during the search (in this case, a 1080ti). It is unclear how well the searched architecture generalizes to different chips like ARM-based mobile processors or specialized AI accelerators.
  • Dataset Diversity: The paper focuses exclusively on Cityscapes. Testing on more diverse datasets (like medical imaging or indoor robotics) would prove the robustness of the IRNN-based attention module.
  • Preprocessing Overhead: There is a notable gap between "inference speed" (166 FPS) and "actual segmentation speed" (48 FPS), suggesting that data loading and post-processing remain significant bottlenecks in a full system pipeline.

Summary updated

Abstract

Image semantic segmentation technology is one of the key technologies for intelligent systems to understand natural scenes. As one of the important research directions in the field of visual intelligence, this technology has broad application scenarios in the fields of mobile robots, drones, smart driving, and smart security. However, in the actual application of mobile robots, problems such as inaccurate segmentation semantic label prediction and loss of edge information of segmented objects and background may occur. This paper proposes an improved structure of a semantic segmentation network based on a deep learning network that combines self-attention neural network and neural network architecture search methods. First, a neural network search method NAS (Neural Architecture Search) is used to find a semantic segmentation network with multiple resolution branches. In the search process, combine the self-attention network structure module to adjust the searched neural network structure, and then combine the semantic segmentation network searched by different branches to form a fast semantic segmentation network structure, and input the picture into the network structure to get the final forecast result. The experimental results on the Cityscapes dataset show that the accuracy of the algorithm is 69.8%, and the segmentation speed is 48/s. It achieves a good balance between real-time and accuracy, can optimize edge segmentation, and has a better performance in complex scenes. Good robustness is suitable for practical application.
BibTeX
@misc{wu2021m,
  author = {Junjun Wu and Huiyu Kuang and Qinghua Lu and Zeqin Lin and Qingwu Shi and Xilin Liu and Xiaoman Zhu},
  title = {{M-FasterSeg: An Efficient Semantic Segmentation Network Based on Neural Architecture Search}},
  year = {2021},
  eprint = {2112.07918},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2112.07918},
}
Figure