Abstract
Most semantic segmentation models treat every video frame as an isolated puzzle, ignoring the inherent temporal consistency of the physical world. This paper introduces a "prior fusion network" that leverages images from the immediate past (temporal priors) to significantly boost segmentation accuracy—especially for challenging dynamic objects. By testing several fusion strategies, the authors demonstrate that a decoder-level integration achieves state-of-the-art performance with a fraction of the parameters required by heavy-duty models like FCN-8.
Problem Statement
In the context of autonomous driving, semantic segmentation is a critical task for safe navigation. While humans naturally utilize memory of recent observations to understand their surroundings, many AI models process images in isolation. This approach is computationally inefficient and ignores valuable context: if a vehicle was present in a specific location one second ago, it is highly likely to remain in the vicinity.
The primary challenge lies in the "messiness" of temporal priors. While static elements like roads and buildings remain constant, dynamic objects such as cars and pedestrians move, and ego-motion shifts the camera's perspective. Simple label propagation is insufficient; a sophisticated method is required to fuse information from a previous "prior" frame with the current input.
Proposed Methods
The researchers developed a prior fusion framework based on an encoder-decoder architecture. They evaluated three distinct strategies for incorporating a prior image captured approximately one second earlier:
- Stacked Prior: A baseline approach where the current and prior images are concatenated as a multi-channel input.
- Embedding Prior: Both images are processed through a shared encoder. Their features are fused at the bottleneck using a learned weighted sum with a tanh activation function.
- Decoder Prior: The most advanced iteration, where features from the prior image are fused with the current image's features at every level of the decoder, rather than just the bottleneck.
By sharing weights between the encoders for the prior and current image branches, the model maintains a low total parameter count, which is essential for real-time applications.
Datasets and Evaluation
The models were trained and evaluated on the CamVid road scene dataset, which consists of 367 training and 233 test images. The dataset features driving sequences with object-class semantic labels (e.g., car, pedestrian, sidewalk) processed at a resolution of 360 x 480.
Key Insights
- Dynamic Object Improvement: Counter-intuitively, the most significant gains were observed in dynamic classes (cars, pedestrians) rather than static ones. The prior appears to provide a "structural anchor" for the scene layout, helping the model isolate and identify moving objects more effectively.
- The Importance of Multi-Level Fusion: The "Decoder Prior" model outperformed the "Embedding Prior," suggesting that both high-level semantic context and low-level spatial details from the past are vital for current frame understanding.
- Efficiency and Compactness: The Decoder Prior model achieved a mean Intersection-over-Union (IoU) of 57.0, matching the performance of the well-known FCN-8 model while utilizing 5x fewer parameters (26.5M vs. 134.5M).
- Learned Propagation: Unlike methods relying on complex optical flow calculations, this approach learns to propagate information directly from raw data, reducing computational overhead during inference.
Results and Qualitative Analysis
The "Decoder Prior" model improved global accuracy for dynamic classes from 69.1% to 73.3% compared to a baseline without priors. Static class accuracy also saw a modest increase from 88.2% to 89.1%.
Qualitatively, the use of a prior helps "clean up" noisy segmentations, resulting in sharper boundaries for fine-grained features like street signs and pedestrians, as shown in the visual comparisons.
Limitations and Future Directions
The authors noted that the CamVid dataset is relatively small and suggested that pre-training on larger datasets like ImageNet could further enhance performance. Additionally, the model is sensitive to the time window; using frames from more than a second in the past degrades performance as the scene changes too significantly. Future work may explore "Scene Graphs" as a more advanced prior or the use of optical flow during training to facilitate unsupervised learning.
Summary updated