Non-differentiable Reward Optimization for Diffusion-based Autonomous Motion Planning

Giwon Lee, Daehee Park, Jaewoo Jeong, Kuk-Jin Yoon

TL;DR

This paper introduces a reinforcement learning (RL) framework to train diffusion models for autonomous motion planning, enabling them to directly optimize non-differentiable objectives like safety and goal achievement. By employing a dynamic thresholding algorithm to shape sparse rewards, the approach significantly improves planning performance and safety compared to existing methods.

What Problem Does This Paper Address?

Autonomous robots, especially in dynamic environments like self-driving cars or navigating crowded spaces, need to plan safe and efficient paths. Diffusion models have shown great promise in predicting complex, multi-modal future trajectories of surrounding agents. However, a critical challenge arises because the most important metrics for successful motion planning – such as avoiding collisions (safety) and reaching a destination (effectiveness) – are inherently non-differentiable. This means standard deep learning training methods, which rely on gradients derived from differentiable loss functions, cannot directly optimize these crucial objectives. Existing approaches often resort to differentiable approximations or indirect methods, which can lead to suboptimal performance.

Methods

The core contribution is a reinforcement learning-based training scheme for diffusion motion planning models. Instead of relying solely on maximum likelihood estimation (which focuses on data distribution), the authors frame the diffusion process as a Markov Decision Process (MDP). They then use policy gradient methods to optimize a reward signal that directly incorporates non-differentiable objectives.

A key innovation is the dynamic thresholding algorithm designed to tackle the problem of sparse rewards. In motion planning, successful outcomes (like avoiding a collision or reaching a goal) might be infrequent within a training batch, leading to weak learning signals. This algorithm adaptively adjusts thresholds for rewards, ensuring that even in sparse scenarios, meaningful signals are extracted to guide the learning process. The reward signal is shaped to keep the mean reward close to a target value (B/2), promoting stable learning.

Datasets

The paper evaluates its approach on two widely used pedestrian trajectory datasets:

  • CrowdNav: Features one designated ego agent and five surrounding agents.
  • ETH-UCY: Used for both validation and training, with past and future trajectory timespans specified.

Key Insights and Ideas

  • Directly optimizing non-differentiable objectives is crucial: The paper strongly argues that for real-world motion planning, directly optimizing metrics like collision rate and success rate is far more effective than relying on differentiable approximations.
  • RL is a powerful tool for diffusion models in planning: By framing diffusion as an MDP, RL allows diffusion models to learn from task-specific rewards, unlocking their potential beyond just data distribution modeling.
  • Dynamic thresholding is key to overcoming sparse rewards: The proposed method's ability to adapt reward thresholds is critical for stable and effective training in scenarios where desired outcomes are infrequent.
  • Non-differentiable rewards can lead to superior performance: Experiments show that optimizing directly with non-differentiable metrics often yields better results than using their differentiable counterparts.

Key Results

  • State-of-the-art performance: The proposed method achieves the best overall performance across safety (collision rate), effectiveness (success rate), and conventional metrics (ADE) on both CrowdNav and ETH-UCY datasets.
  • Significant collision rate reduction: On CrowdNav, the collision rate was reduced from 0.0628 to 0.0150, outperforming baselines that use collision cost functions.
  • Improved safety and effectiveness: Qualitative results demonstrate the model's ability to generate collision-free trajectories and improve goal-reaching capabilities, even in dense, complex scenes.
  • Flexibility in reward design: The framework can incorporate multiple objectives simultaneously, allowing for a balanced optimization of safety, effectiveness, and comfort.

Limitations and Future Work

  • Limitations: The paper acknowledges that while their method improves performance, it still relies on a pre-trained forecaster for some baselines. The dynamic thresholding algorithm's effectiveness is demonstrated, but its theoretical guarantees and optimal parameter tuning for diverse scenarios might require further investigation.
  • Future Work: The authors suggest that their approach is broadly applicable to other non-differentiable objectives in motion planning, such as comfort or efficiency, and can be extended to more complex multi-agent scenarios. The flexibility of incorporating arbitrary reward types is highlighted as a direction for future exploration.

Summary updated

Abstract

Safe and effective motion planning is crucial for autonomous robots. Diffusion models excel at capturing complex agent interactions, a fundamental aspect of decision-making in dynamic environments. Recent studies have successfully applied diffusion models to motion planning, demonstrating their competence in handling complex scenarios and accurately predicting multi-modal future trajectories. Despite their effectiveness, diffusion models have limitations in training objectives, as they approximate data distributions rather than explicitly capturing the underlying decision-making dynamics. However, the crux of motion planning lies in non-differentiable downstream objectives, such as safety (collision avoidance) and effectiveness (goal-reaching), which conventional learning algorithms cannot directly optimize. In this paper, we propose a reinforcement learning-based training scheme for diffusion motion planning models, enabling them to effectively learn non-differentiable objectives that explicitly measure safety and effectiveness. Specifically, we introduce a reward-weighted dynamic thresholding algorithm to shape a dense reward signal, facilitating more effective training and outperforming models trained with differentiable objectives. State-of-the-art performance on pedestrian datasets (CrowdNav, ETH-UCY) compared to various baselines demonstrates the versatility of our approach for safe and effective motion planning.
BibTeX
@inproceedings{lee2025non,
  author = {Giwon Lee and Daehee Park and Jaewoo Jeong and Kuk-Jin Yoon},
  title = {{Non-differentiable Reward Optimization for Diffusion-based Autonomous Motion Planning}},
  booktitle = {IEEE/RSJ International Conference on Intelligent Robots and Systems},
  year = {2025},
  url = {https://openreview.net/forum?id=vz8OKNitUg},
}

Loading related content...

Figure