VRT-Net: Real-Time Scene Parsing via Variable Resolution Transform

Jogendra Nath Kundu, Gaurav Singh Rajput, R. Venkatesh BabuWACV 2020 · OpenReview

Abstract

Urban scene parsing is a basic requirement for various autonomous navigation systems especially in self-driving. Most of the available approaches employ generic image parsing architectures designed for segmentation of object focused scene captured in indoor setups. However, images captured in car-mounted cameras exhibit an extreme effect of perspective geometry, causing a significant scale disparity between near and farther objects. Recognizing this, we formalize a unique Variable Resolution Transform (VRT) technique motivated from the foveal magnification in human eye. Following this, we design a Fovea Estimation Network (FEN) which is trained to estimate a single most convenient fixation location along with the associated magnification factor, best suited for a given input image. The proposed framework is designed to enable its usage as a wrapper over the available real-time scene parsing models, thereby demonstrating a superior trade-off between speed and quality as compared to the prior state-of-the-arts.

Introduction

In autonomous driving, scene parsing—the task of assigning semantic categories like 'car', 'road', or 'pedestrian' to every pixel—is a safety-critical requirement. However, urban images present a unique challenge due to extreme perspective geometry. Objects near the camera appear large and detailed, while distant objects are tiny and often cluttered together. Standard deep learning models frequently fail in two ways: blurring distant objects into "merged regions" or losing global context for large nearby objects, resulting in "splitted regions."

The Variable Resolution Transform (VRT)

To address these scale disparities without the computational cost of high-resolution processing, VRT-Net introduces a bio-mimetic approach inspired by the human eye.

Visualizing the Variable Resolution Transform (VRT) mechanism, which magnifies distant areas of interest while compressing the periphery to normalize object scales.

The core of the system is the Variable Resolution Transform (VRT). This non-linear mapping magnifies a specific "fovea point" (typically near the horizon) while mathematically compressing peripheral regions. A lightweight Fovea Estimation Network (FEN) predicts the optimal fovea location and magnification factor for each frame, reusing feature-extraction layers from the main model to maintain efficiency.

Methodology: A Wrapper Approach

VRT-Net is designed as a "wrapper" that can enhance existing real-time segmentation models. It employs a dual-branch processing strategy:

  1. Original Branch: Processes the standard input to maintain peripheral context.
  2. Transformed Branch: Processes the VRT-warped image to capture fine details of distant objects.

The results are fused using Perspective-Aware Aggregation, which applies a spatial weight map to prioritize the transformed branch for foveal details and the original branch for the periphery. Crucially, the VRT transform is mathematically invertible, allowing predictions to be mapped back to the original image coordinates perfectly.

Experimental Results

The framework was evaluated on the Cityscapes and CamVid datasets, using ICNet and ThunderNet as baseline real-time models.

Accuracy (mIoU%) vs. Speed (Frames/Second) trade-off. VRT-Net (pink stars) significantly improves accuracy over baselines like ICNet and ThunderNet while maintaining real-time performance.

As shown in the performance analysis, adding the VRT wrapper to ICNet boosted the Mean Intersection over Union (mIoU) on Cityscapes from 70.6% to 75.2% while maintaining a real-time speed of 29.4 FPS. When applied to the high-speed ThunderNet, the system achieved 90.9 FPS with a 69.1% mIoU. These results demonstrate a superior trade-off between speed and quality compared to prior state-of-the-art methods like PSPNet or DeepLab, which are significantly slower.

Conclusion and Future Work

VRT-Net provides a computationally efficient way to handle scale disparity in urban scenes. By prioritizing important regions through foveal magnification, it enables lightweight models to achieve high accuracy on small, distant hazards. While the current model estimates a single fovea point, future work may explore multi-fovea estimation to handle complex scenes with multiple simultaneous points of interest, such as pedestrians on both sides of a vehicle's path.

Summary updated

Abstract

Urban scene parsing is a basic requirement for various autonomous navigation systems especially in self-driving. Most of the available approaches employ generic image parsing architectures designed for segmentation of object focused scene captured in indoor setups. However, images captured in car-mounted cameras exhibit an extreme effect of perspective geometry, causing a significant scale disparity between near and farther objects. Recognizing this, we formalize a unique Variable Resolution Transform (VRT) technique motivated from the foveal magnification in human eye. Following this, we design a Fovea Estimation Network (FEN) which is trained to estimate a single most convenient fixation location along with the associated magnification factor, best suited for a given input image. The proposed framework is designed to enable its usage as a wrapper over the available real-time scene parsing models, thereby demonstrating a superior trade-off between speed and quality as compared to the prior state-of-the-arts.
BibTeX
@inproceedings{kundu2020vrt,
  author = {Jogendra Nath Kundu and Gaurav Singh Rajput and R. Venkatesh Babu},
  title = {{VRT-Net: Real-Time Scene Parsing via Variable Resolution Transform}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2020},
  url = {https://openreview.net/forum?id=bYH3W2ic6i1},
}
Figure