TL;DR
The paper introduces PrefCVAE, a framework that uses weak preference pairs to make generative trajectory prediction models (like CVAEs) semantically controllable. By structuring the latent space through relative rankings rather than absolute ground-truth labels, PrefCVAE allows autonomous driving planners to explicitly simulate specific behavioral attributes (such as target velocities) without degrading baseline prediction accuracy.
What Problem Does This Paper Address?
Predicting how surrounding vehicles and pedestrians will move is a cornerstone of safe autonomous driving. Deep generative models, particularly Conditional Variational Autoencoders (CVAEs), excel at capturing the multimodal nature of human behavior by generating a diverse set of potential future trajectories.
However, standard CVAEs suffer from a major limitation: their latent spaces are implicit and unstructured. They generate diverse paths randomly, meaning downstream motion planners cannot easily request specific types of trajectories (e.g., "show me what happens if the adjacent vehicle accelerates aggressively"). For safe and proactive ego-vehicle planning, having controllable diversity—where trajectories can be generated based on predefined semantic attributes—is far more valuable than purely random variations.
Methods
The authors propose PrefCVAE (Preference CVAE), a weakly supervised augmentation for existing CVAE architectures.
Key components of the method include:
- Factorized Latent Space: The latent space is split into two parts: semantically meaningful latents ($z_S$) dedicated to specific attributes, and implicit latents ($z_I$) to capture remaining unstructured noise.
- Beta Distribution Latents: Instead of the standard Gaussian distribution, PrefCVAE utilizes a Beta distribution for the latent space, providing an explicit, bounded domain ($[0, 1]$) that maps cleanly to bounded physical metrics.
- Pairwise Preference Loss: To train the semantic latent space without needing explicit ground-truth latent values, the framework uses a ranking-based approach:
- It samples two distinct semantic latent values ($z_S^0 < z_S^1$).
- The decoder rolls out two corresponding auxiliary trajectory predictions ($\hat{y}^0$ and $\hat{y}^1$).
- An oracle program evaluates both trajectories using a differentiable metric $m(\cdot)$ (e.g., average velocity).
- A soft, cross-entropy-based preference loss penalizes the model if the ranking of the generated trajectories' metrics does not align monotonically with the ranking of the sampled latents.
Additionally, the authors introduce a hyperparameter called the use rate ($\nu$), which dictates the probability that a collected preference pair is actually utilized during training, allowing them to evaluate the model's data efficiency.
Datasets
The framework was evaluated using the large-scale nuScenes autonomous driving dataset. The authors modified the popular Transformer-based CVAE model, AgentFormer, into a bounded variant called $\beta$-AgentFormer to serve as their baseline and integration target.
Key Insights and Ideas
- Supervision via Ranking is Enough: The most compelling takeaway is that you do not need exact, absolute semantic labels to structure a continuous latent space. Simply telling the model which of two generated trajectories exhibits "more" of a trait is sufficient to enforce a smooth, monotonic mapping.
- Less Data Can Be More (The Use Rate Paradox): Surprisingly, using 100% of the preference pairs ($\nu = 1.0$) during training did not yield the best performance. Dropping pairs randomly ($\nu = 0.25$) actually improved the model's violation rate and alignment robustness. The authors hypothesize that partial labeling acts as a regularizer, preventing the model from overfitting to stochastic noise inherent in the preference pairs.
- Model-Agnostic and Agent-Independent: Cross-configuration tests revealed that the learned semantic latents are highly robust. Controlling the latent factor of just a single vehicle in a multi-agent scene alters only that specific vehicle's behavior, proving the latents are learned independently per agent.
Key Results
- Maintained Baseline Accuracy: PrefCVAE achieves controllability without sacrificing prediction quality. The minimum Average Displacement Error ($\text{minADE}_5$) remained virtually unchanged (2.64 meters for PrefCVAE vs. 2.62 meters for the base model).
- Monotonic Control: As shown in the latent space traversal tests, sweeping $z_S$ from 0.1 to 0.9 steadily increased the average velocity of the predicted trajectories in a predictable, monotonic fashion.
- Low Violation Rates: At a use rate of $\nu = 0.25$, the Agent-Wise Violation Rate (measuring how often the generated velocity trend contradicted the latent assignment) dropped to an impressive 0.0090%.
Limitations and Future Work
- Simple Semantic Metrics: The current evaluation relies solely on a straightforward, easily differentiable oracle metric: average velocity.
- Single-Dimension Scaling: The framework was only tested using a single semantic dimension. Scaling PrefCVAE to handle multiple simultaneous attributes (e.g., combining velocity control with lane-changing aggressiveness) remains a challenge due to potential correlations and entanglements between latent factors.
- Future Directions: The authors point out the need to explore non-differentiable or human-provided preferences using techniques like Gumbel-Softmax, alongside more complex social metrics like Social Value Orientation (SVO).
Summary updated