Introduction
In the domain of autonomous driving, spatio-temporal alignment is a critical process that links current sensor observations with historical data. By understanding where an object was and how it was moving, a perception system can maintain accurate tracks even when objects are partially occluded or sensor data is degraded.
However, current end-to-end (E2E) perception models often rely on oversimplified assumptions, such as constant velocity models, or purely implicit "black box" neural attention mechanisms. These approaches frequently fail in complex real-world scenarios involving rapid braking, sharp turns, or environmental noise like heavy snow. To address these limitations, we introduce HAT (Multiple Hypotheses Spatio-Temporal Alignment), a plug-and-play module designed to adaptively select from a library of physical motion hypotheses.
The HAT Framework
HAT moves away from the "one-size-fits-all" approach by explicitly modeling diverse motion patterns through two primary modules: the Temporal Alignment Module and the Spatial Alignment Module.
Temporal Alignment Module
At the core of HAT is a Motion Model Library (MML). This library contains five well-established physical models: Constant Velocity (CV), STATIC, Constant Acceleration (CA), Constant Turn Rate and Velocity (CTRV), and Constant Turn Rate and Acceleration (CTRA). For every historical object, the system generates five distinct hypotheses regarding its current state. To handle unobservable states like acceleration and yaw rate, HAT utilizes a Multi-Layer Perceptron (MLP) to infer these values from historical position and velocity data.
Spatial Alignment Module
Rather than manually tuning which model to use, HAT employs an adaptive decoder to act as a "judge." This module analyzes the object's internal feature representation (the "query") and learns to assign weights to the five hypotheses. For instance, if a query suggests a vehicle is entering a curve, the CTRV or CTRA models receive higher weights. These weighted hypotheses are then fused into a single, optimized alignment proposal. To ensure physical plausibility, a stability constraint prevents the model from making erratic jumps that deviate too far from the grounded physical models.
Experimental Results and Insights
The researchers evaluated HAT using the nuScenes dataset and its corrupted counterpart, nuScenes-C, which simulates adverse weather conditions.
Key Performance Metrics
HAT demonstrated significant improvements across various state-of-the-art baselines:
- Tracking Accuracy: Integrated with the DETR3D detector, HAT achieved a new high of 46.0% AMOTA on the nuScenes test set.
- Safety and Planning: In end-to-end driving simulations (SparseDrive-HAT), the module reduced the collision rate by 32% compared to the baseline.
- Environmental Robustness: On the nuScenes-C "Snow" benchmark, HAT improved detection (NDS) by 5.0% and tracking (AMOTA) by 4.9%. This suggests that when visual semantics are blurred by weather, explicit physical motion cues become the primary source of reliable information.
Efficiency and Deployment
Despite the complexity of maintaining multiple hypotheses, HAT is highly efficient. It adds approximately 7ms of latency per frame, making it suitable for real-time deployment in autonomous vehicle stacks.
Conclusion
HAT proves that explicit physical modeling still plays a vital role in modern neural perception systems. By allowing a network to "self-select" the most appropriate physical model for each object on the fly, HAT provides a robust safety net that improves both tracking accuracy and planning safety. Future work will explore expanding the Motion Model Library to include specialized behaviors, such as non-linear pedestrian movement, and tighter integration with downstream path-planning modules.
Summary updated