Abstract
An environment representation (ER) is a substantial part of every autonomous system. It introduces a common interface between perception and other system components, such as decision making, and allows downstream algorithms to deal with abstracted data without knowledge of the used sensor. In this work, we propose and evaluate a novel architecture that generates an egocentric, grid-based, predictive, and semantically-interpretable ER. In particular, we provide a proof of concept for the spatio-temporal fusion of multiple camera sequences and short-term prediction in such an ER. Our design utilizes a strong semantic segmentation network together with depth and egomotion estimates to first extract semantic information from multiple camera streams and then transform these separately into egocentric temporally-aligned bird's-eye view grids. A deep encoder-decoder network is trained to fuse a stack of these grids into a unified semantic grid representation and to predict the dynamics of its surrounding. We evaluate this representation on real-world sequences of the Cityscapes dataset and show that our architecture can make accurate predictions in complex sensor fusion scenarios and significantly outperforms a model-driven baseline in a category-based evaluation.
Introduction
Autonomous vehicles rely on an Environment Representation (ER) to bridge the gap between raw sensor data (perception) and path planning (decision making). However, standard perception systems face three major hurdles: sensor dependency, where segmentation is tied to specific camera perspectives; fusion complexity, involving the mathematical tax of combining overlapping views; and system latency, where processing time renders the world model obsolete by the time it is generated. This paper proposes a Semantic Grid—an egocentric, grid-based bird's-eye view (BEV) map—that provides a unified, interpretable, and predictive interface for downstream algorithms.
Methodology
The proposed pipeline consists of two main stages designed to transform raw image data into a predictive BEV map.
Stage I: Spatial Transformation and Alignment
First, a DeepLabv3 network performs semantic segmentation on raw RGB images. Using depth information from stereo cameras and known camera intrinsics/extrinsics, these semantic labels are projected onto a 2D ground plane. Finally, grids from different time steps are aligned to a common future time point ($\tau$) using the vehicle's egomotion, ensuring that static objects like roads and buildings line up across frames.
Stage II: Predictive Fusion
The aligned grids are stacked and processed by a deep Encoder-Decoder (ED) CNN. This network performs two critical tasks: Fusion, merging data from different cameras and resolving spatial ambiguities; and Prediction, estimating the future positions of dynamic objects like cars and pedestrians. A masked loss function is employed to ignore unobserved areas while rewarding the network for "remembering" objects that were visible in the past but are currently occluded.
Experimental Evaluation
The researchers evaluated the model using the Cityscapes dataset. To test multi-camera fusion robustness, they simulated "Sensor Splits" by cropping images into separate sections with artificial blind spots. The dataset's 30-frame sequences at 17 Hz provided the necessary temporal data for training the predictive elements.
Key Results and Insights
The proposed architecture significantly outperformed model-driven baselines, particularly for dynamic objects. In the default configuration, the model achieved an Intersection over Union (IoU) of 0.61 for cars, compared to 0.52 for the baseline. Even with simulated blind spots, the system maintained high accuracy for static classes like "road" (0.81 IoU), demonstrating its ability to "hallucinate" missing information based on context.
Beyond raw metrics, the model exhibited several intelligent behaviors:
- Implicit Egomotion Learning: The network reached high performance even without explicit translational movement data, learning to estimate motion from shifts in the static environment.
- Occlusion Handling: The model demonstrates a "memory" effect, predicting the likely position of a car in the target frame even if it becomes occluded in the intermediate frames.
- Interpretability: Unlike "black box" models, the semantic grid is human-readable, allowing engineers to visualize exactly what the car perceives and where it expects objects to move.
Conclusion and Future Work
While the semantic grid offers a robust and interpretable ER, limitations remain. The model struggles with very small objects like traffic signs due to spatial discretization errors. Future work will focus on adaptive latency estimation, expanding the semantic classes (e.g., pedestrian poses), and integrating multi-modal data from LiDAR or Radar to complement camera-derived semantics.
Summary updated