Abstract
Semantic segmentation algorithms that can robustly segment objects across multiple camera viewpoints are crucial for navigation and safety in autonomous driving. While existing algorithms treat each image in isolation, autonomous vehicles often revisit the same locations. We propose leveraging this a priori knowledge to improve semantic segmentation of images from sequential driving datasets. We examine several methods to fuse these temporal scene priors and introduce a prior fusion network that learns how to transfer this information. Our model improves the accuracy of dynamic object classes from 69.1% to 73.3%, and static classes from 88.2% to 89.1%.
Introduction
Semantic segmentation—the task of labeling every pixel in an image—is a cornerstone of autonomous driving. However, most state-of-the-art models treat every video frame as an isolated event, ignoring the fact that vehicles often move through environments where the scene changes very little from one second to the next. This "a priori" knowledge is usually ignored because of the difficulty in effectively combining a past image (the prior) with the current one. Since objects move and the camera shifts, a simple overlay is insufficient; the model requires a sophisticated mechanism to determine which information from the past remains relevant.
Methodology
Building upon the SegNet encoder-decoder architecture, we introduced a Prior Fusion Network that takes two inputs: the current frame and a "prior" frame captured one second earlier. We explored two primary architectures for this fusion:
- Embedding Prior: Features from both images are fused only at the "bottleneck," where the image is at its most abstract, low-resolution representation.
- Decoder Prior: Features are fused at every level of the decoder, allowing the model to incorporate prior knowledge at multiple scales, from high-level concepts to fine-grained spatial details.
The fusion is handled by a learned module (Module A) that utilizes a weighted sum and a tanh activation function, a technique borrowed from Recurrent Neural Networks (RNNs) to effectively gate and transfer information.
Experimental Results
The study utilized the CamVid (Cambridge-driving Labeled Video Database) dataset. Our analysis revealed that spatial resolution is critical; fusing information only at the bottleneck (Embedding Prior) actually decreased class accuracy and Intersection-over-Union (IoU). This suggests that while the bottleneck captures "what" is in the scene, the decoder levels are necessary to provide the "where."
The Decoder Prior model emerged as the superior architecture, outperforming the baseline across all metrics. Notably, the biggest gains were seen in dynamic objects (pedestrians, cars, cyclists), which saw a jump from 69.1% to 73.3%. The prior provides a contextual hint that helps the model suppress noise and better delineate smaller, moving objects that are often misclassified in single-frame analysis.
Conclusion
By incorporating temporal priors, we achieved a significant improvement in segmentation accuracy, particularly for dynamic objects. Future work will explore using multiple frames across a longer time window and more advanced representations, such as scene graphs, to further refine the fusion process and improve the robustness of autonomous navigation systems.
Summary updated