FarSee-Net: Real-Time Semantic Segmentation by Efficient Multi-scale Context Aggregation and Feature Space Super-resolution

Zhanpeng Zhang, Kaipeng Zhang

Abstract

Real-time semantic segmentation is critical for robotic applications with limited computational resources. A primary challenge lies in managing object scale variations and leveraging contextual information within a tight computation budget. This paper introduces FarSee-Net, which utilizes a novel Cascaded Factorized Atrous Spatial Pyramid Pooling (CF-ASPP) module for efficient context aggregation. Furthermore, we rethink the up-sampling process as a super-resolution task, employing optimized sub-pixel convolutions to improve accuracy, particularly for sub-sampled inputs. FarSee-Net achieves a superior latency-accuracy trade-off, reaching 68.4% mIoU at 84 FPS on the Cityscapes test set using a single NVIDIA Titan X (Maxwell) GPU.

The Challenge of Real-Time Segmentation

Semantic segmentation—assigning a class label to every pixel—is inherently computationally intensive. For autonomous systems, this must occur in real-time, facing two significant hurdles:

  1. Scale Variation: Objects appear at vastly different sizes. Capturing "context" (spatial relationships) typically requires heavy modules like Atrous Spatial Pyramid Pooling (ASPP), which are often too slow for real-time deployment.
  2. Resolution Loss: To maintain speed, networks often aggressively downsample features. Standard non-parametric up-sampling (e.g., bilinear interpolation) frequently fails to recover sharp boundaries or small objects.

Proposed Methodology: FarSee-Net

FarSee-Net addresses these issues through a front-end/back-end architecture designed for efficiency.

Overview of the FarSee-Net architecture featuring the CF-ASPP module.

Cascaded Factorized ASPP (CF-ASPP)

Standard ASPP uses parallel $3 \times 3$ atrous convolutions. FarSee-Net "factorizes" these into a $1 \times 1$ point-wise convolution and a $3 \times 3$ depth-wise atrous convolution, reducing computation by approximately 8.8x. By cascading two of these lightweight modules, the network achieves a larger receptive field and better context aggregation without the heavy computational cost of a single large ASPP.

Feature Space Super-resolution

Instead of simple interpolation, FarSee-Net treats the final up-sampling as a learned Super-resolution (SR) task. Using sub-pixel convolutions (pixel shuffling), the network learns to reconstruct high-resolution details from low-resolution feature maps. This approach significantly improves the recovery of thin structures and small, distant objects.

Key Results and Performance

FarSee-Net demonstrates a state-of-the-art balance between speed and accuracy compared to existing real-time models.

Accuracy vs. Speed trade-off on Cityscapes dataset.
  • Efficiency: On the Cityscapes test set, the model achieves 68.4% mIoU at 84 FPS (with 512x768 input).
  • Hardware Versatility: When deployed on embedded hardware like the Jetson AGX Xavier, it maintains a functional 21.2 FPS, making it viable for actual robotic platforms.
  • Ablation Insights: Replacing standard ASPP with CF-ASPP simultaneously improved accuracy and reduced latency, proving the effectiveness of the factorized design.

Conclusion

FarSee-Net provides a robust framework for real-time semantic segmentation by optimizing context aggregation and up-sampling. While highly efficient, future work could involve further hardware-specific optimizations using libraries like TensorRT or exploring even lighter backbone architectures to push the boundaries of performance on ultra-low-power devices.

Summary updated

Abstract

Real-time semantic segmentation is desirable in many robotic applications with limited computation resources. One challenge of semantic segmentation is to deal with the object scale variations and leverage the context. How to perform multi-scale context aggregation within limited computation budget is important. In this paper, firstly, we introduce a novel and efficient module called Cascaded Factorized Atrous Spatial Pyramid Pooling (CF-ASPP). It is a lightweight cascaded structure for Convolutional Neural Networks (CNNs) to efficiently leverage context information. On the other hand, for runtime efficiency, state-of-the-art methods will quickly decrease the spatial size of the inputs or feature maps in the early network stages. The final high-resolution result is usually obtained by non-parametric up-sampling operation (e.g. bilinear interpolation). Differently, we rethink this pipeline and treat it as a super-resolution process. We use optimized super-resolution operation in the up-sampling step and improve the accuracy, especially in sub-sampled input image scenario for real-time applications. By fusing the above two improvements, our methods provide better latency-accuracy trade-off than the other state-of-the-art methods. In particular, we achieve 68.4% mIoU at 84 fps on the Cityscapes test set with a single Nivida Titan X (Maxwell) GPU card. The proposed module can be plugged into any feature extraction CNN and benefits from the CNN structure development.
BibTeX
@inproceedings{zhang2020farsee,
  author = {Zhanpeng Zhang and Kaipeng Zhang},
  title = {{FarSee-Net: Real-Time Semantic Segmentation by Efficient Multi-scale Context Aggregation and Feature Space Super-resolution}},
  booktitle = {IEEE International Conference on Robotics and Automation},
  year = {2020},
  url = {https://openreview.net/forum?id=FHeoJ85ZkU},
}
Figure