Rethinking the Spatio-Temporal Alignment of End-to-End 3D Perception

Xiaoyu Li, Peidong Li, Xian Wu, Long Shi, Dedong Liu, Yitao Wu, Jiajia Fu, Dixiao Cui, Lijun Zhao, Lining SunarXiv 2025 · DOI · arXiv

Introduction

In the domain of autonomous driving, spatio-temporal alignment is a critical process that links current sensor observations with historical data. By understanding where an object was and how it was moving, a perception system can maintain accurate tracks even when objects are partially occluded or sensor data is degraded.

However, current end-to-end (E2E) perception models often rely on oversimplified assumptions, such as constant velocity models, or purely implicit "black box" neural attention mechanisms. These approaches frequently fail in complex real-world scenarios involving rapid braking, sharp turns, or environmental noise like heavy snow. To address these limitations, we introduce HAT (Multiple Hypotheses Spatio-Temporal Alignment), a plug-and-play module designed to adaptively select from a library of physical motion hypotheses.

The HAT Framework

HAT moves away from the "one-size-fits-all" approach by explicitly modeling diverse motion patterns through two primary modules: the Temporal Alignment Module and the Spatial Alignment Module.

The HAT architecture consists of a Temporal Alignment Module that generates multiple motion hypotheses and a Spatial Alignment Module that adaptively fuses them based on object queries.

Temporal Alignment Module

At the core of HAT is a Motion Model Library (MML). This library contains five well-established physical models: Constant Velocity (CV), STATIC, Constant Acceleration (CA), Constant Turn Rate and Velocity (CTRV), and Constant Turn Rate and Acceleration (CTRA). For every historical object, the system generates five distinct hypotheses regarding its current state. To handle unobservable states like acceleration and yaw rate, HAT utilizes a Multi-Layer Perceptron (MLP) to infer these values from historical position and velocity data.

Spatial Alignment Module

Rather than manually tuning which model to use, HAT employs an adaptive decoder to act as a "judge." This module analyzes the object's internal feature representation (the "query") and learns to assign weights to the five hypotheses. For instance, if a query suggests a vehicle is entering a curve, the CTRV or CTRA models receive higher weights. These weighted hypotheses are then fused into a single, optimized alignment proposal. To ensure physical plausibility, a stability constraint prevents the model from making erratic jumps that deviate too far from the grounded physical models.

Experimental Results and Insights

The researchers evaluated HAT using the nuScenes dataset and its corrupted counterpart, nuScenes-C, which simulates adverse weather conditions.

Key Performance Metrics

HAT demonstrated significant improvements across various state-of-the-art baselines:

  • Tracking Accuracy: Integrated with the DETR3D detector, HAT achieved a new high of 46.0% AMOTA on the nuScenes test set.
  • Safety and Planning: In end-to-end driving simulations (SparseDrive-HAT), the module reduced the collision rate by 32% compared to the baseline.
  • Environmental Robustness: On the nuScenes-C "Snow" benchmark, HAT improved detection (NDS) by 5.0% and tracking (AMOTA) by 4.9%. This suggests that when visual semantics are blurred by weather, explicit physical motion cues become the primary source of reliable information.
Qualitative results showing how HAT adaptively assigns probabilities to different motion models (e.g., STATIC vs. Non-Static) across different frames to maintain accurate tracking.

Efficiency and Deployment

Despite the complexity of maintaining multiple hypotheses, HAT is highly efficient. It adds approximately 7ms of latency per frame, making it suitable for real-time deployment in autonomous vehicle stacks.

Conclusion

HAT proves that explicit physical modeling still plays a vital role in modern neural perception systems. By allowing a network to "self-select" the most appropriate physical model for each object on the fly, HAT provides a robust safety net that improves both tracking accuracy and planning safety. Future work will explore expanding the Motion Model Library to include specialized behaviors, such as non-linear pedestrian movement, and tighter integration with downstream path-planning modules.

Summary updated

Abstract

Spatio-temporal alignment is crucial for temporal modeling of end-to-end (E2E) perception in autonomous driving (AD), providing valuable structural and textural prior information. Existing methods typically rely on the attention mechanism to align objects across frames, simplifying the motion model with a unified explicit physical model (constant velocity, etc.). These approaches prefer semantic features for implicit alignment, challenging the importance of explicit motion modeling in the traditional perception paradigm. However, variations in motion states and object features across categories and frames render this alignment suboptimal. To address this, we propose HAT, a spatio-temporal alignment module that allows each object to adaptively decode the optimal alignment proposal from multiple hypotheses without direct supervision. Specifically, HAT first utilizes multiple explicit motion models to generate spatial anchors and motion-aware feature proposals for historical instances. It then performs multi-hypothesis decoding by incorporating semantic and motion cues embedded in cached object queries, ultimately providing the optimal alignment proposal for the target frame. On nuScenes, HAT consistently improves 3D temporal detectors and trackers across diverse baselines. It achieves state-of-the-art tracking results with 46.0% AMOTA on the test set when paired with the DETR3D detector. In an object-centric E2E AD method, HAT enhances perception accuracy (+1.3% mAP, +3.1% AMOTA) and reduces the collision rate by 32%. When semantics are corrupted (nuScenes-C), the enhancement of motion modeling by HAT enables more robust perception and planning in the E2E AD.
BibTeX
@misc{li2025rethinking,
  author = {Xiaoyu Li and Peidong Li and Xian Wu and Long Shi and Dedong Liu and Yitao Wu and Jiajia Fu and Dixiao Cui and Lijun Zhao and Lining Sun},
  title = {{Rethinking the Spatio-Temporal Alignment of End-to-End 3D Perception}},
  year = {2025},
  eprint = {2512.23635},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2512.23635},
}
Figure