Direct Post-Training Preference Alignment for Multi-Agent Motion Generation Models Using Implicit Feedback from Pre-training Demonstrations

Ran Tian, Kratarth GoelarXiv 2025 · DOI · arXiv

Abstract

Recent advances in embodied AI have reframed multi-agent motion generation as a sequence modeling task. However, standard supervised learning often fails to capture nuanced human preferences, and traditional Reinforcement Learning from Human Feedback (RLHF) is prohibitively expensive for complex traffic simulations. We present DPA-OMF (Direct Preference Alignment from Occupancy Measure Matching Feedback), a novel framework that leverages existing expert demonstrations to automatically rank model-generated behaviors. By utilizing Optimal Transport to measure semantic distances between simulations and expert data, DPA-OMF enables effective post-training alignment without manual labeling. Our results show that a 1-million parameter model fine-tuned with DPA-OMF achieves realism scores comparable to state-of-the-art models 100 times its size.

Introduction

The adoption of "next-token prediction" architectures for autonomous driving and robotics has significantly improved motion forecasting. Despite technical accuracy in trajectory prediction, these models frequently produce behaviors that are jerky, unrealistic, or prone to near-misses in dense multi-agent environments. Aligning these models with human preferences is the standard solution in other domains, but ranking multi-agent scenes—often involving over 100 interacting agents—is exceptionally difficult and costly for human annotators.

Existing "Alignment from Demonstrations" (AFD) methods typically rely on a binary adversarial assumption: expert data is "good," and all model-generated data is "bad." This approach ignores the rich signal present in the model's own varying attempts and can lead to spurious correlations where the model optimizes for simple features like smoothness while ignoring critical safety constraints.

Methods: Direct Preference Alignment from Occupancy Measure Matching Feedback (DPA-OMF)

DPA-OMF addresses the alignment bottleneck by treating pre-training expert data as an implicit judge. The pipeline consists of four primary stages:

Overview of the DPA-OMF pipeline, showing model pre-training followed by direct preference alignment using occupancy measure matching feedback.
  1. Occupancy Measure Matching: Instead of raw coordinates, the system analyzes "occupancy measures"—the distribution of semantic features (e.g., velocity, proximity to road boundaries, and collision status) visited by agents over time.
  2. Optimal Transport (OT) for Preference Distance: To quantify the quality of a generated rollout, the authors compute the OT distance between the rollout's occupancy measure and that of the corresponding expert demonstration. This measures the "work" required to transform model behavior into expert behavior in a high-dimensional feature space.
  3. Implicit Ranking: For any given scene, the model generates multiple rollouts. These are ranked based on their OT distance to the expert demonstration. Rollouts closest to the expert are labeled as "preferred," while those furthest away are "unpreferred."
  4. Contrastive Learning: These automated rankings facilitate fine-tuning via a multi-agent variant of Contrastive Preference Learning (CPL). This process pushes the model's policy toward preferred behaviors and away from undesirable ones like road-boundary violations.
Illustration of rollout ranking based on distance to ground truth expert demonstrations using optimal transport on occupancy measures.

Experimental Results

The framework was evaluated using the Waymo Open Motion Dataset (WOMD), specifically the Waymo Open Sim Agents Challenge (WOSAC), which involves up to 128 interacting agents.

Performance and Efficiency

A compact 1M parameter model aligned with DPA-OMF achieved a Composite Realism score of 0.739. This performance is on par with significantly larger state-of-the-art models, such as Trajeglish (35M parameters) and SMART (102M parameters). This suggests that preference alignment can effectively compensate for smaller model capacities by focusing the learning signal on high-quality behaviors.

Comparison to Baselines

DPA-OMF consistently outperformed standard Supervised Fine-Tuning (SFT) and traditional adversarial alignment. While SFT on "best-of-N" samples often failed to improve the realism metric, the contrastive nature of DPA-OMF successfully penalized "unpreferred" behaviors. Furthermore, the automated ranking process represents a massive reduction in human effort; the authors estimate that generating the equivalent preference data manually would require over 600 days of continuous human labor.

Discussion and Future Work

DPA-OMF demonstrates that expert demonstrations contain untapped "implicit preferences" that can be recycled for post-training alignment at zero additional annotation cost. However, the current reliance on manually engineered features for the preference distance suggests a potential area for improvement: learning these features directly from data. Additionally, while the 1M parameter model is highly efficient, applying DPA-OMF to 100M+ parameter architectures could potentially set new benchmarks in multi-agent realism.

Summary updated

BibTeX
@misc{tian2025direct,
  author = {Ran Tian and Kratarth Goel},
  title = {{Direct Post-Training Preference Alignment for Multi-Agent Motion Generation Models Using Implicit Feedback from Pre-training Demonstrations}},
  year = {2025},
  eprint = {2503.20105},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2503.20105},
}
Figure