FarSee-Net: Real-Time Semantic Segmentation by Efficient Multi-scale Context Aggregation and Feature Space Super-resolution

Zhanpeng Zhang, Kaipeng Zhang

TL;DR

FarSee-Net is a lightweight deep learning framework designed for real-time semantic segmentation on resource-constrained platforms like robots. By introducing a cascaded, factorized version of Atrous Spatial Pyramid Pooling (ASPP) and treating the final upsampling step as a learned super-resolution task, the authors achieve an impressive 84 FPS with 68.4% mIoU on the Cityscapes dataset using a single NVIDIA Titan X GPU.

What Problem Does This Paper Address?

Semantic segmentation—the task of assigning a class label to every pixel in an image—is computationally expensive. For autonomous robots and vehicles, this task must happen in real-time. However, two major hurdles exist:

  1. Scale Variation: Objects appear at different sizes. Capturing "context" (the relationship between pixels at different distances) usually requires heavy modules like ASPP, which are too slow for real-time use.
  2. Resolution Loss: To save speed, most networks aggressively downsample images. They then use simple, non-parametric methods like bilinear interpolation to blow the image back up to its original size, which often results in blurry boundaries and lost details for small objects.

FarSee-Net aims to bridge this "latency-accuracy trade-off" by making context aggregation more efficient and upsampling more "intelligent."

Methods

The authors propose two primary innovations within a front-end/back-end architecture:

  • Cascaded Factorized ASPP (CF-ASPP): Standard ASPP uses multiple parallel $3 \times 3$ atrous (dilated) convolutions to capture multi-scale context. FarSee-Net "factorizes" these into a $1 \times 1$ point-wise convolution (to handle channel interaction) followed by a $3 \times 3$ depth-wise atrous convolution (to handle spatial context). This reduces computation by roughly 8.8x. They then cascade two of these lightweight modules, allowing the network to see even more context without the massive computational hit of a single large ASPP module.
  • Feature Space Super-resolution: Instead of using standard bilinear interpolation to resize the low-resolution feature maps back to the input size, the authors treat this as a Super-resolution (SR) problem. They use sub-pixel convolutions (also known as pixel shuffling). This allows the network to learn how to best reconstruct high-resolution details from low-resolution features, significantly improving the recovery of thin and small objects.

The system uses a standard ResNet-18 as a backbone (front-end) and plugs their CF-ASPP and SR modules into the back-end.

Datasets

The model was trained and evaluated on the Cityscapes dataset, a standard benchmark for urban scene understanding. It contains high-resolution (1024x2048) images of street scenes with fine annotations for 19 classes (roads, cars, pedestrians, etc.).

Key Insights and Ideas

  • Factorization is a Free Lunch: The paper demonstrates that the same factorization tricks used to speed up standard convolutions (like in MobileNet) work exceptionally well for atrous convolutions used in context aggregation.
  • Cascading > Scaling: Rather than making one context module wider or deeper, cascading two smaller, factorized modules provides a better balance of receptive field size and speed.
  • Upsampling is Reconstruction: The "aha moment" here is rethinking the final layer of a segmentation network. By viewing the final upscaling not as a mathematical resizing but as a learned reconstruction (Super-resolution), the model recovers much sharper object boundaries than its competitors.

Key Results

FarSee-Net achieves a superior balance of speed and accuracy compared to several well-known real-time models:

  • Performance: 68.4% mIoU at 84 FPS on the Cityscapes test set (using 512x768 input).
  • Efficiency: When tested on an embedded Jetson AGX Xavier, it maintained a playable 21.2 FPS, proving its utility for actual robotic hardware.
  • Ablation Success: The authors showed that switching from standard ASPP to their CF-ASPP improved accuracy while reducing latency, a rare win-win in deep learning.
  • Qualitative Edge: Visual results show that FarSee-Net is particularly good at identifying occluded cars and thin structures (like poles or distant pedestrians) that other real-time networks often "blur" away.

Limitations and Future Work

While FarSee-Net is highly efficient, the authors note that there is still room for improvement:

  • Hardware Optimization: The current implementation uses standard PyTorch; the authors suggest that using specialized libraries like TensorRT or NVDLA could further boost the FPS on embedded systems.
  • Model Compression: The paper focuses on architectural efficiency but suggests that combining their approach with network pruning and quantization could make the model even leaner for ultra-low-power devices.
  • Backbone Dependency: While the back-end is novel, the performance still relies heavily on the pre-trained front-end (ResNet-18). Exploring even lighter backbones could be a future direction.

Summary updated

Abstract

Real-time semantic segmentation is desirable in many robotic applications with limited computation resources. One challenge of semantic segmentation is to deal with the object scale variations and leverage the context. How to perform multi-scale context aggregation within limited computation budget is important. In this paper, firstly, we introduce a novel and efficient module called Cascaded Factorized Atrous Spatial Pyramid Pooling (CF-ASPP). It is a lightweight cas-caded structure for Convolutional Neural Networks (CNNs) to efficiently leverage context information. On the other hand, for runtime efficiency, state-of-the-art methods will quickly decrease the spatial size of the inputs or feature maps in the early network stages. The final high-resolution result is usually obtained by non-parametric up-sampling operation (e.g. bilinear interpolation). Differently, we rethink this pipeline and treat it as a super-resolution process. We use optimized super-resolution operation in the up-sampling step and improve the accuracy, especially in sub-sampled input image scenario for real-time applications. By fusing the above two improvements, our methods provide better latency-accuracy trade-off than the other state-of-the-art methods. In particular, we achieve 68.4% mIoU at 84 fps on the Cityscapes test set with a single Nivida Titan X (Maxwell) GPU card. The proposed module can be plugged into any feature extraction CNN and benefits from the CNN structure development.
BibTeX
@inproceedings{zhang2020farsee,
  author = {Zhanpeng Zhang and Kaipeng Zhang},
  title = {{FarSee-Net: Real-Time Semantic Segmentation by Efficient Multi-scale Context Aggregation and Feature Space Super-resolution}},
  booktitle = {IEEE International Conference on Robotics and Automation},
  year = {2020},
  url = {https://openreview.net/forum?id=FHeoJ85ZkU},
}

Loading related content...

Figure