Abstract
Autonomous vehicle (AV) safety relies heavily on trajectory prediction modules to forecast the movement of surrounding agents. However, these modules are susceptible to adversarial attacks where small, human-imperceptible perturbations in a vehicle's history can lead to dangerous prediction errors. We propose Semi-supervised Semantics-guided Adversarial Training (SSAT), a novel defense mechanism that disentangles raw trajectory data into high-level semantic features. By anchoring predictions in stable semantic intents—such as lateral direction and longitudinal headway—SSAT significantly improves model resilience. Experimental results on industry-standard benchmarks show that SSAT reduces prediction errors under attack by up to 73% while maintaining competitive performance in benign scenarios.
Introduction: The Vulnerability of Trajectory Prediction
In the autonomous driving stack, the prediction module forecasts the future paths of nearby vehicles based on their historical movements. Recent studies have identified this module as a critical vulnerability. A malicious actor can execute "wiggly" driving patterns that, while appearing normal to a human observer, trick a victim AV's prediction model into forecasting a false trajectory. Such "Attacked predictions" can trigger unnecessary emergency braking or, worse, lead to collisions.
Defending these models is inherently more complex than defending image classifiers. Unlike discrete image labels, trajectory prediction is a time-series regression problem involving continuous coordinates and high stochasticity. Standard adversarial training (AT) often fails to generalize across the diverse and unseen attack patterns possible in the physical world.
Methods: Semantics-guided Defense
The core of the SSAT approach is the transition from raw coordinate processing to semantic-level reasoning. The authors utilize a Semi-supervised Adversarial Autoencoder (AAE) to map inputs into a disentangled latent space.
SSAT Architecture and Semantic Disentanglement
The SSAT framework decomposes the latent space into three distinct components:
- Lateral Features: Categorized into discrete intentions (e.g., moving forward, turning left, or turning right).
- Longitudinal Features: Modeled using "time headway" (the gap between vehicles) following a log-normal distribution.
- Remaining Latent Space: Captures general noise and context via a Gaussian distribution.
By forcing the model to focus on these high-level semantics, the defense creates an "anchor." Even if an attacker perturbs the input coordinates, the underlying semantic intent (e.g., "the car is slowing down") remains stable, preventing the entire prediction from collapsing.
Semi-supervised Adversarial Training
Since labeling every driving scenario is impractical, SSAT employs a semi-supervised approach. It uses domain knowledge to automatically label clear behaviors in the training set, guiding the adversarial training process. During training, the model is exposed to Projected Gradient Descent (PGD) attacks, and the loss function is optimized to ensure that perturbed inputs still map to the correct semantic latent vectors. To mitigate the "robustness-accuracy trade-off," the authors incorporate MixUp techniques, blending adversarial and benign samples to maintain high performance in normal driving conditions.
Experimental Evaluation
The researchers evaluated SSAT on three major benchmarks: Argoverse 1, Argoverse 2, and ApolloScape. These datasets provide hundreds of thousands of real-world driving scenarios across diverse urban environments.
Key Results
- Significant Error Reduction: SSAT reduced Average Displacement Error (ADE) under attack by 32% to 73% across the tested datasets.
- Superior Generalization: Unlike standard AT, which often only defends against the specific attack seen during training, SSAT demonstrated robust generalization. Training on one attack type (e.g., ADE-based) provided protection against entirely different patterns (e.g., lateral shifts).
- Stability: The AAE architecture ensures that if an attack only targets one dimension (e.g., lateral), the other dimensions (e.g., longitudinal) remain unaffected, preserving partial prediction accuracy.
Conclusion and Future Directions
SSAT represents a significant step toward securing the "brain" of autonomous vehicles. By leveraging high-level semantics as a defensive anchor, the model moves beyond the fragility of raw data regression. While the current implementation uses basic semantic labels (left/right/forward), future work aims to incorporate more complex interactions, such as yielding or pedestrian dynamics. Additionally, testing these robust predictions in closed-loop simulations will be vital to understanding their full impact on AV planning and control safety.
Summary updated