Abstract
Urban scene parsing faces significant challenges due to scale heterogeneity, where objects vary drastically in size based on their distance from the camera. We present FoveaNet, a deep learning framework that estimates a dense perspective heatmap to "undo" perspective distortion. By employing a specialized "fovea" branch to focus on distant, small-scale objects and a perspective-aware CRF to smooth results adaptively, FoveaNet achieves state-of-the-art performance on Cityscapes and CamVid benchmarks.
Introduction
In autonomous driving, semantic segmentation models must accurately identify objects ranging from massive nearby buses to tiny distant pedestrians. Standard Convolutional Neural Networks (CNNs) often struggle with this "scale heterogeneity." Large objects are frequently fragmented into multiple classes (known as "broken-down" errors), while small, distant objects are misclassified or lose boundary definition. FoveaNet addresses these issues by mimicking the human eye's fovea, providing sharp central vision for distant regions while maintaining global context.
Methodology
The FoveaNet architecture integrates perspective geometry directly into the parsing process through three core components.
Perspective Estimation Network (PEN)
Instead of identifying a single vanishing point, the PEN generates a dense, one-channel heatmap. Higher values indicate regions where small, distant objects are likely to cluster. This heatmap serves as a geometric prior for the subsequent stages.
Perspective-aware Parsing Network
This component utilizes a two-branch Fully Convolutional Network (FCN):
- Coarse Branch: Processes the full image to capture global scene context.
- Fovea Branch: Uses the PEN heatmap to identify a "fovea region." It crops and zooms into this area, effectively normalizing the scale of distant objects before parsing. The outputs of both branches are then fused.
Perspective-aware Dense CRFs
To refine the segmentation, FoveaNet employs a novel Conditional Random Field (CRF) model. It uses the perspective heatmap to apply adaptive smoothing: lighter smoothing for small objects to preserve detail, and stronger smoothing for large objects to correct fragmentation. It also incorporates object detection boxes from Faster R-CNN to maintain boundary integrity.
Experimental Results
FoveaNet was evaluated on the Cityscapes and CamVid datasets, demonstrating significant improvements over established baselines like DeepLabv2.
On Cityscapes, the model achieved a Mean IoU of 74.1%. Notably, it showed a 5.2% improvement in instance-level IoU (iIoU), highlighting its effectiveness on small-scale instances. Qualitative analysis confirmed a substantial reduction in "broken-down" errors for large vehicles and up to a 6% IoU gain for categories like poles and traffic signs. On CamVid, FoveaNet set a new state-of-the-art with 93.3% global accuracy.
Discussion and Conclusion
The primary innovation of FoveaNet is the use of perspective geometry for scale normalization. By "zooming in" on the vanishing point, the network processes distant objects at a scale comparable to nearby ones. While the model introduces some computational overhead due to the PEN and object detection steps, its ability to handle scale heterogeneity provides a robust framework for safer urban scene understanding. Future work may involve tighter integration of depth estimation to further refine the perspective priors.
Summary updated