Using Image Priors to Improve Scene Understanding

Brigit Schroeder, Hanlin Tang, Alexandre AlahiarXiv 2019 · arXiv

Abstract

Most semantic segmentation models treat every video frame as an isolated puzzle, ignoring the inherent temporal consistency of the physical world. This paper introduces a "prior fusion network" that leverages images from the immediate past (temporal priors) to significantly boost segmentation accuracy—especially for challenging dynamic objects. By testing several fusion strategies, the authors demonstrate that a decoder-level integration achieves state-of-the-art performance with a fraction of the parameters required by heavy-duty models like FCN-8.

Problem Statement

In the context of autonomous driving, semantic segmentation is a critical task for safe navigation. While humans naturally utilize memory of recent observations to understand their surroundings, many AI models process images in isolation. This approach is computationally inefficient and ignores valuable context: if a vehicle was present in a specific location one second ago, it is highly likely to remain in the vicinity.

The primary challenge lies in the "messiness" of temporal priors. While static elements like roads and buildings remain constant, dynamic objects such as cars and pedestrians move, and ego-motion shifts the camera's perspective. Simple label propagation is insufficient; a sophisticated method is required to fuse information from a previous "prior" frame with the current input.

Proposed Methods

The researchers developed a prior fusion framework based on an encoder-decoder architecture. They evaluated three distinct strategies for incorporating a prior image captured approximately one second earlier:

  1. Stacked Prior: A baseline approach where the current and prior images are concatenated as a multi-channel input.
  2. Embedding Prior: Both images are processed through a shared encoder. Their features are fused at the bottleneck using a learned weighted sum with a tanh activation function.
  3. Decoder Prior: The most advanced iteration, where features from the prior image are fused with the current image's features at every level of the decoder, rather than just the bottleneck.
Architectural diagrams for embedding-level and decoder-level fusion strategies.

By sharing weights between the encoders for the prior and current image branches, the model maintains a low total parameter count, which is essential for real-time applications.

Datasets and Evaluation

The models were trained and evaluated on the CamVid road scene dataset, which consists of 367 training and 233 test images. The dataset features driving sequences with object-class semantic labels (e.g., car, pedestrian, sidewalk) processed at a resolution of 360 x 480.

Key Insights

  • Dynamic Object Improvement: Counter-intuitively, the most significant gains were observed in dynamic classes (cars, pedestrians) rather than static ones. The prior appears to provide a "structural anchor" for the scene layout, helping the model isolate and identify moving objects more effectively.
  • The Importance of Multi-Level Fusion: The "Decoder Prior" model outperformed the "Embedding Prior," suggesting that both high-level semantic context and low-level spatial details from the past are vital for current frame understanding.
  • Efficiency and Compactness: The Decoder Prior model achieved a mean Intersection-over-Union (IoU) of 57.0, matching the performance of the well-known FCN-8 model while utilizing 5x fewer parameters (26.5M vs. 134.5M).
  • Learned Propagation: Unlike methods relying on complex optical flow calculations, this approach learns to propagate information directly from raw data, reducing computational overhead during inference.

Results and Qualitative Analysis

The "Decoder Prior" model improved global accuracy for dynamic classes from 69.1% to 73.3% compared to a baseline without priors. Static class accuracy also saw a modest increase from 88.2% to 89.1%.

Qualitative comparison showing how the inclusion of a prior image improves segmentation results over the baseline.

Qualitatively, the use of a prior helps "clean up" noisy segmentations, resulting in sharper boundaries for fine-grained features like street signs and pedestrians, as shown in the visual comparisons.

Limitations and Future Directions

The authors noted that the CamVid dataset is relatively small and suggested that pre-training on larger datasets like ImageNet could further enhance performance. Additionally, the model is sensitive to the time window; using frames from more than a second in the past degrades performance as the scene changes too significantly. Future work may explore "Scene Graphs" as a more advanced prior or the use of optical flow during training to facilitate unsupervised learning.

Summary updated

Abstract

Semantic segmentation algorithms that can robustly segment objects across multiple camera viewpoints are crucial for assuring navigation and safety in emerging applications such as autonomous driving. Existing algorithms treat each image in isolation, but autonomous vehicles often revisit the same locations or maintain information from the immediate past. We propose a simple yet effective method for leveraging these image priors to improve semantic segmentation of images from sequential driving datasets. We examine several methods to fuse these temporal scene priors, and introduce a prior fusion network that is able to learn how to transfer this information. The prior fusion model improves the accuracy over the non-prior baseline from 69.1% to 73.3% for dynamic classes, and from 88.2% to 89.1% for static classes. Compared to models such as FCN-8, our prior method achieves the same accuracy with 5 times fewer parameters. We used a simple encoder decoder backbone, but this general prior fusion method could be applied to more complex semantic segmentation backbones. We also discuss how structured representations of scenes in the form of a scene graph could be leveraged as priors to further improve scene understanding.
BibTeX
@misc{schroeder2019using,
  author = {Brigit Schroeder and Hanlin Tang and Alexandre Alahi},
  title = {{Using Image Priors to Improve Scene Understanding}},
  year = {2019},
  eprint = {1910.01198},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1910.01198},
}
Figure