FoveaNet: Perspective-aware Urban Scene Parsing

Xin Li, Zequn Jie, Wei Wang, Changsong Liu, Jimei Yang, Xiaohui Shen, Zhe Lin, Qiang Chen, Shuicheng Yan, Jiashi Feng

Abstract

Urban scene parsing faces significant challenges due to scale heterogeneity, where objects vary drastically in size based on their distance from the camera. We present FoveaNet, a deep learning framework that estimates a dense perspective heatmap to "undo" perspective distortion. By employing a specialized "fovea" branch to focus on distant, small-scale objects and a perspective-aware CRF to smooth results adaptively, FoveaNet achieves state-of-the-art performance on Cityscapes and CamVid benchmarks.

Introduction

In autonomous driving, semantic segmentation models must accurately identify objects ranging from massive nearby buses to tiny distant pedestrians. Standard Convolutional Neural Networks (CNNs) often struggle with this "scale heterogeneity." Large objects are frequently fragmented into multiple classes (known as "broken-down" errors), while small, distant objects are misclassified or lose boundary definition. FoveaNet addresses these issues by mimicking the human eye's fovea, providing sharp central vision for distant regions while maintaining global context.

Methodology

The FoveaNet architecture integrates perspective geometry directly into the parsing process through three core components.

The FoveaNet architecture consists of a Perspective Estimation Network (PEN), a two-branch parsing network, and perspective-aware CRFs.

Perspective Estimation Network (PEN)

Instead of identifying a single vanishing point, the PEN generates a dense, one-channel heatmap. Higher values indicate regions where small, distant objects are likely to cluster. This heatmap serves as a geometric prior for the subsequent stages.

Perspective-aware Parsing Network

This component utilizes a two-branch Fully Convolutional Network (FCN):

  • Coarse Branch: Processes the full image to capture global scene context.
  • Fovea Branch: Uses the PEN heatmap to identify a "fovea region." It crops and zooms into this area, effectively normalizing the scale of distant objects before parsing. The outputs of both branches are then fused.

Perspective-aware Dense CRFs

To refine the segmentation, FoveaNet employs a novel Conditional Random Field (CRF) model. It uses the perspective heatmap to apply adaptive smoothing: lighter smoothing for small objects to preserve detail, and stronger smoothing for large objects to correct fragmentation. It also incorporates object detection boxes from Faster R-CNN to maintain boundary integrity.

Experimental Results

FoveaNet was evaluated on the Cityscapes and CamVid datasets, demonstrating significant improvements over established baselines like DeepLabv2.

Qualitative results showing the fovea region selection and the resulting segmentation improvements for both large and small objects.

On Cityscapes, the model achieved a Mean IoU of 74.1%. Notably, it showed a 5.2% improvement in instance-level IoU (iIoU), highlighting its effectiveness on small-scale instances. Qualitative analysis confirmed a substantial reduction in "broken-down" errors for large vehicles and up to a 6% IoU gain for categories like poles and traffic signs. On CamVid, FoveaNet set a new state-of-the-art with 93.3% global accuracy.

Discussion and Conclusion

The primary innovation of FoveaNet is the use of perspective geometry for scale normalization. By "zooming in" on the vanishing point, the network processes distant objects at a scale comparable to nearby ones. While the model introduces some computational overhead due to the PEN and object detection steps, its ability to handle scale heterogeneity provides a robust framework for safer urban scene understanding. Future work may involve tighter integration of depth estimation to further refine the perspective priors.

Summary updated

BibTeX
@inproceedings{li2017foveanet,
  author = {Xin Li and Zequn Jie and Wei Wang and Changsong Liu and Jimei Yang and Xiaohui Shen and Zhe Lin and Qiang Chen and Shuicheng Yan and Jiashi Feng},
  title = {{FoveaNet: Perspective-aware Urban Scene Parsing}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2017},
  doi = {10.1109/iccv.2017.91},
}
Figure