Abstract
Real-time semantic segmentation is critical for robotic applications with limited computational resources. A primary challenge lies in managing object scale variations and leveraging contextual information within a tight computation budget. This paper introduces FarSee-Net, which utilizes a novel Cascaded Factorized Atrous Spatial Pyramid Pooling (CF-ASPP) module for efficient context aggregation. Furthermore, we rethink the up-sampling process as a super-resolution task, employing optimized sub-pixel convolutions to improve accuracy, particularly for sub-sampled inputs. FarSee-Net achieves a superior latency-accuracy trade-off, reaching 68.4% mIoU at 84 FPS on the Cityscapes test set using a single NVIDIA Titan X (Maxwell) GPU.
The Challenge of Real-Time Segmentation
Semantic segmentation—assigning a class label to every pixel—is inherently computationally intensive. For autonomous systems, this must occur in real-time, facing two significant hurdles:
- Scale Variation: Objects appear at vastly different sizes. Capturing "context" (spatial relationships) typically requires heavy modules like Atrous Spatial Pyramid Pooling (ASPP), which are often too slow for real-time deployment.
- Resolution Loss: To maintain speed, networks often aggressively downsample features. Standard non-parametric up-sampling (e.g., bilinear interpolation) frequently fails to recover sharp boundaries or small objects.
Proposed Methodology: FarSee-Net
FarSee-Net addresses these issues through a front-end/back-end architecture designed for efficiency.
Cascaded Factorized ASPP (CF-ASPP)
Standard ASPP uses parallel $3 \times 3$ atrous convolutions. FarSee-Net "factorizes" these into a $1 \times 1$ point-wise convolution and a $3 \times 3$ depth-wise atrous convolution, reducing computation by approximately 8.8x. By cascading two of these lightweight modules, the network achieves a larger receptive field and better context aggregation without the heavy computational cost of a single large ASPP.
Feature Space Super-resolution
Instead of simple interpolation, FarSee-Net treats the final up-sampling as a learned Super-resolution (SR) task. Using sub-pixel convolutions (pixel shuffling), the network learns to reconstruct high-resolution details from low-resolution feature maps. This approach significantly improves the recovery of thin structures and small, distant objects.
Key Results and Performance
FarSee-Net demonstrates a state-of-the-art balance between speed and accuracy compared to existing real-time models.
- Efficiency: On the Cityscapes test set, the model achieves 68.4% mIoU at 84 FPS (with 512x768 input).
- Hardware Versatility: When deployed on embedded hardware like the Jetson AGX Xavier, it maintains a functional 21.2 FPS, making it viable for actual robotic platforms.
- Ablation Insights: Replacing standard ASPP with CF-ASPP simultaneously improved accuracy and reduced latency, proving the effectiveness of the factorized design.
Conclusion
FarSee-Net provides a robust framework for real-time semantic segmentation by optimizing context aggregation and up-sampling. While highly efficient, future work could involve further hardware-specific optimizations using libraries like TensorRT or exploring even lighter backbone architectures to push the boundaries of performance on ultra-low-power devices.
Summary updated