Controllable Generative Trajectory Prediction via Weak Preference Alignment

Yongxi Cao, Julian F. Schumann, Jens Kober, Joni Pajarinen, Arkady Zgonnikov

TL;DR

The paper introduces PrefCVAE, a framework that uses weak preference pairs to make generative trajectory prediction models (like CVAEs) semantically controllable. By structuring the latent space through relative rankings rather than absolute ground-truth labels, PrefCVAE allows autonomous driving planners to explicitly simulate specific behavioral attributes (such as target velocities) without degrading baseline prediction accuracy.

What Problem Does This Paper Address?

Predicting how surrounding vehicles and pedestrians will move is a cornerstone of safe autonomous driving. Deep generative models, particularly Conditional Variational Autoencoders (CVAEs), excel at capturing the multimodal nature of human behavior by generating a diverse set of potential future trajectories.

However, standard CVAEs suffer from a major limitation: their latent spaces are implicit and unstructured. They generate diverse paths randomly, meaning downstream motion planners cannot easily request specific types of trajectories (e.g., "show me what happens if the adjacent vehicle accelerates aggressively"). For safe and proactive ego-vehicle planning, having controllable diversity—where trajectories can be generated based on predefined semantic attributes—is far more valuable than purely random variations.

Motivation for controllable prediction in autonomous driving planning

Methods

The authors propose PrefCVAE (Preference CVAE), a weakly supervised augmentation for existing CVAE architectures.

Key components of the method include:

  1. Factorized Latent Space: The latent space is split into two parts: semantically meaningful latents ($z_S$) dedicated to specific attributes, and implicit latents ($z_I$) to capture remaining unstructured noise.
  2. Beta Distribution Latents: Instead of the standard Gaussian distribution, PrefCVAE utilizes a Beta distribution for the latent space, providing an explicit, bounded domain ($[0, 1]$) that maps cleanly to bounded physical metrics.
  3. Pairwise Preference Loss: To train the semantic latent space without needing explicit ground-truth latent values, the framework uses a ranking-based approach:
  4. It samples two distinct semantic latent values ($z_S^0 < z_S^1$).
  5. The decoder rolls out two corresponding auxiliary trajectory predictions ($\hat{y}^0$ and $\hat{y}^1$).
  6. An oracle program evaluates both trajectories using a differentiable metric $m(\cdot)$ (e.g., average velocity).
  7. A soft, cross-entropy-based preference loss penalizes the model if the ranking of the generated trajectories' metrics does not align monotonically with the ranking of the sampled latents.

Additionally, the authors introduce a hyperparameter called the use rate ($\nu$), which dictates the probability that a collected preference pair is actually utilized during training, allowing them to evaluate the model's data efficiency.

Figure 2: Requirements of controllable predic- tion: (1) Encoder enforces the latent distribu- tion to estimate a predefined semantic factor m(·); (2) Latent assignments generate seman- tically controllable predictions (sampling line colors correspond to trajectories on the right).

Datasets

The framework was evaluated using the large-scale nuScenes autonomous driving dataset. The authors modified the popular Transformer-based CVAE model, AgentFormer, into a bounded variant called $\beta$-AgentFormer to serve as their baseline and integration target.

Key Insights and Ideas

  • Supervision via Ranking is Enough: The most compelling takeaway is that you do not need exact, absolute semantic labels to structure a continuous latent space. Simply telling the model which of two generated trajectories exhibits "more" of a trait is sufficient to enforce a smooth, monotonic mapping.
  • Less Data Can Be More (The Use Rate Paradox): Surprisingly, using 100% of the preference pairs ($\nu = 1.0$) during training did not yield the best performance. Dropping pairs randomly ($\nu = 0.25$) actually improved the model's violation rate and alignment robustness. The authors hypothesize that partial labeling acts as a regularizer, preventing the model from overfitting to stochastic noise inherent in the preference pairs.
  • Model-Agnostic and Agent-Independent: Cross-configuration tests revealed that the learned semantic latents are highly robust. Controlling the latent factor of just a single vehicle in a multi-agent scene alters only that specific vehicle's behavior, proving the latents are learned independently per agent.
Figure 3: (a). Controllable predictions, manifesting utility of PrefCVAE decoder. For each column, Up: prediction visualization; Middle: semantic metric w.r.t. z value (horizontal dashed lines are ground truth values); Below: ADE (solid)/FDE (dashed). PrefCVAE can control the prediction: For model trained with PrefCVAE, larger z value always leads to larger average velocity, as learned with the preference loss. Also noticeably, the best accuracy occurs around the latent values that pertain to the ground truth velocity (the dashed horizontal lines). (b). Test-set-regressed distributions of controlled latent factor. Each color pertains to trajectory predicted with a different z value. The ideal result should resemble 9 Dirac delta distributions with modes at each ground truth z.

Key Results

  • Maintained Baseline Accuracy: PrefCVAE achieves controllability without sacrificing prediction quality. The minimum Average Displacement Error ($\text{minADE}_5$) remained virtually unchanged (2.64 meters for PrefCVAE vs. 2.62 meters for the base model).
  • Monotonic Control: As shown in the latent space traversal tests, sweeping $z_S$ from 0.1 to 0.9 steadily increased the average velocity of the predicted trajectories in a predictable, monotonic fashion.
  • Low Violation Rates: At a use rate of $\nu = 0.25$, the Agent-Wise Violation Rate (measuring how often the generated velocity trend contradicted the latent assignment) dropped to an impressive 0.0090%.

Limitations and Future Work

  • Simple Semantic Metrics: The current evaluation relies solely on a straightforward, easily differentiable oracle metric: average velocity.
  • Single-Dimension Scaling: The framework was only tested using a single semantic dimension. Scaling PrefCVAE to handle multiple simultaneous attributes (e.g., combining velocity control with lane-changing aggressiveness) remains a challenge due to potential correlations and entanglements between latent factors.
  • Future Directions: The authors point out the need to explore non-differentiable or human-provided preferences using techniques like Gumbel-Softmax, alongside more complex social metrics like Social Value Orientation (SVO).

Summary updated

Abstract

Deep generative models such as conditional variational autoencoders (CVAEs) have shown great promise for predicting trajectories of surrounding agents in autonomous vehicle planning. State-of-the-art models have achieved remarkable accuracy in such prediction tasks. Besides accuracy, diversity is also crucial for safe planning because human behaviors are inherently uncertain and multimodal. However, existing methods generally lack a scheme to generate controllably diverse trajectories, which is arguably more useful than randomly diversified trajectories, to the end of safe planning. To address this, we propose PrefCVAE, an augmented CVAE framework that uses weakly labeled preference pairs to imbue latent variables with semantic attributes. Using average velocity as an example attribute, we demonstrate that PrefCVAE enables controllable, semantically meaningful predictions without degrading baseline accuracy. Our results show the effectiveness of preference supervision as a cost-effective way to enhance sampling-based generative models.
BibTeX
@misc{cao2025controllable,
  author = {Yongxi Cao and Julian F. Schumann and Jens Kober and Joni Pajarinen and Arkady Zgonnikov},
  title = {{Controllable Generative Trajectory Prediction via Weak Preference Alignment}},
  year = {2025},
  eprint = {2510.10731},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2510.10731},
}
Figure