Abstract
Occupancy maps are widely recognized as an efficient method for facilitating robot motion planning in static environments. However, for intelligent vehicles, occupancy of both the present and future moments is required to ensure safe driving. In the automotive industry, the accurate and continuous prediction of future occupancy maps in traffic scenarios remains a formidable challenge. This paper investigates multi-sensor spatio-temporal fusion strategies for continuous occupancy prediction in a systematic manner. This paper presents FusionMotion, a novel bird's eye view (BEV) occupancy predictor which is capable of achieving the fusion of asynchronous multi-sensor data and predicting the future occupancy map with variable time intervals and temporal horizons. Remarkably, FusionMotion features the adoption of neural ordinary differential equations on recurrent neural networks for occupancy prediction. FusionMotion learns derivatives of BEV features over temporal horizons, updates the implicit sensor's BEV feature measurements and propagates future states for each ODE step. Extensive experiments on large-scale nuScenes and Lyft L5 datasets demonstrate that FusionMotion significantly outperforms previous methods. In addition, it outperforms the BEVFusion-style fusion strategy on the Lyft L5 dataset while reducing synchronization requirements.
Introduction: The Challenge of Temporal Stiffness
In autonomous driving, occupancy forecasting—predicting which parts of the 3D space around a vehicle will be occupied by objects like cars or pedestrians—is critical for safe navigation. However, current state-of-the-art systems suffer from two major "stiffness" problems that limit their real-world utility:
- Strict Synchronization: Existing models typically assume that camera images and LiDAR sweeps arrive simultaneously. In practice, sensors operate at different frame rates with inherent offsets, forcing systems to wait for data alignment, which introduces dangerous latency.
- Discrete Snapshots: Most models are trained to predict the world at fixed intervals (e.g., every 0.5 seconds). This lack of temporal resolution makes it difficult for vehicles to perform split-second maneuvers that require predictions at "in-between" moments.
To address these constraints, we introduce FusionMotion (implemented as the StreamingFlow framework), moving from a "snapshot" mindset to a continuous "streaming" mindset.
Methodology: Continuous-Time Fusion via Neural ODEs
The core innovation of FusionMotion is the SpatialGRU-ODE, which integrates Neural Ordinary Differential Equations (N-ODEs) into a recurrent neural network architecture. The pipeline operates through three primary stages:
- BEV Encoding: Raw data from heterogeneous sensors (cameras and LiDAR) are transformed into a shared Bird’s-Eye-View (BEV) feature space.
- Neural ODE Integration: Instead of standard discrete RNN updates, the model uses a Neural ODE to model the derivative (rate of change) of BEV features over time. This allows the system to represent the environment's evolution as a continuous function.
- Update-Predict-Update Cycle:
- Update: When a new sensor measurement arrives—regardless of timing or sensor type—the model uses a "trusting gate" to fuse this information into the current hidden state.
- Predict: To forecast the future, the model integrates the learned ODE forward. Because it is an ODE, integration can be stopped at any arbitrary timestamp (e.g., 0.1s or 2.0s) to retrieve a prediction.
- Probabilistic Alignment: A Kullback-Leibler (KL) divergence loss is employed to ensure predicted features remain realistic and aligned with actual future observations during training.
Experimental Results
FusionMotion was evaluated on the nuScenes and Lyft L5 datasets, two industry-standard benchmarks for large-scale autonomous driving scenarios.
As shown in the results table, FusionMotion (StreamingFlow) sets new benchmarks across multiple metrics:
- Superior Accuracy: On nuScenes, the base model outperformed the previous best vision-based method (BEVerse) by +7.0 IoU and the best LiDAR-based method (BE-STI) by +7.7 IoU.
- Robustness on Lyft L5: It surpassed the ST-P3 baseline by a staggering +20.6 IoU and +23.5 VPQ (Video Panoptic Quality).
- Efficiency and Scalability: The model runs at approximately 0.19s per sample, significantly faster than competitors like FIERY or StretchBEV (~0.64s). Furthermore, for dense future predictions, the ODE approach is more memory-efficient than standard GRU models, which often suffer from out-of-memory (OOM) errors at high temporal resolutions.
Key Insights and Discussion
Temporal Decoupling: A significant advantage of this approach is that prediction granularity is decoupled from training granularity. The model can be trained on sparse labels (every 0.5s) but generate smooth, high-frequency future forecasts (every 0.05s).
Zero-Shot Generalization: By learning the underlying "flow" of a scene via an ODE, the model generalizes well to unseen time horizons. Models trained for 2-second horizons can reasonably estimate occupancy up to 8 seconds ahead without retraining.
Asynchrony as a Feature: Rather than struggling with misaligned data, FusionMotion treats every data point as a trigger to update its internal state, making it resilient to real-world hardware jitter.
Limitations and Future Work
Despite its performance, the model exhibits gradual performance decay as the forecast moves further into the unseen future. Additionally, increasing prediction frequency for smoother "video" output increases runtime delay, requiring a balance between smoothness and real-time constraints. Future work will explore expanding this continuous-time approach to motion planning and mapping for a more unified autonomous vehicle architecture.
Summary updated