In complex environments—whether a busy city intersection or a cluttered kitchen counter—there is rarely a single "correct" way to move. A self-driving car might pass a stalled vehicle on the left or the right; a robot arm might reach for a mug from above or from the side.
Traditional robotic planning often relies on regression, which attempts to predict a single "average" path. However, in multi-modal scenarios, the average of two valid paths (like going left and going right) is often a path straight through the obstacle. To solve this, a growing body of research has shifted toward generative modeling, particularly Diffusion Models, to represent the full distribution of possible futures rather than a single deterministic guess.
The Failure of the Average Path
The core challenge in trajectory prediction is "mode collapse" or "averaging." As noted by Xingtai Gui and colleagues in TrajDiff, standard regression-based planners struggle when faced with road forks or intersections. If the training data contains examples of drivers turning both left and right, a regression model may predict a dangerous middle path.
Generative frameworks solve this by modeling the probability density of trajectories. Early approaches used Normalizing Flows, as seen in the Trajformer architecture by Jonathan Francis, to map simple distributions to complex, multi-modal path hypotheses. More recently, researchers like Jiachen Li have used optimal transport principles in frameworks like STG-DAT to ensure that predicted trajectories remain physically feasible while capturing diverse potential behaviors.
Planning as Iterative Denoising
The breakthrough in this field is treating motion planning not as a search problem, but as a generative denoising process. In a diffusion-based planner, the model starts with a sequence of random noise (a "nonsense" trajectory) and iteratively refines it into a coherent path conditioned on environmental context, such as semantic maps or LiDAR data.
This approach, exemplified by Trajectory Diffusion (T-diff) by Xinyao Yu, allows the agent to maintain multiple "guesses" simultaneously. Because the model learns the underlying distribution of valid movements, it can naturally represent distinct strategic options. For instance, in highway merging, Jacob Knaup demonstrates that a diffusion-based solver can evaluate both "yielding" and "overtaking" maneuvers without getting trapped in the local minima that plague traditional gradient-based optimizers.
Generation then Evaluation
While diffusion models are excellent at generating diverse paths, an autonomous system must eventually commit to one. This has led to a "generation-then-evaluation" paradigm.
In the Gen-Drive framework, Xinshuo Weng and Zhiyu Huang use a multi-agent diffusion model to propose a variety of socially consistent future scenarios. These candidates are then passed to a reward model—sometimes fine-tuned with feedback from Large Language Models—to select the safest and most comfortable route. Similarly, Wonjeong Ryu proposes SUPER-AD, which evaluates generated paths against a "safety-weighted occupancy grid" to account for perceptual uncertainty in the environment.
Efficiency through Structured Representations
A common critique of diffusion models is their computational cost, as they typically require many iterations to generate a sample. Researchers are addressing this through more efficient mathematical formulations and structured representations:
- Flow Matching: Zebin Xing introduced GoalFlow, which uses Flow Matching to generate trajectories in a single denoising step, making it viable for real-time autonomous driving. Matthew Johnson-Roberson has extended this to $SE(3)$ space for complex robotic manipulation.
- Geometric Priors: Instead of predicting raw coordinates, models like SanD-Planner predict B-spline control points. This ensures the generated paths are inherently smooth and reduces the dimensionality of the generation task.
- Normalized Residuals: The ResAD framework shifts the task from predicting absolute positions to predicting "residuals" relative to a physics-based inertial reference. This centers the data distribution, allowing the model to converge much faster than traditional diffusion planners.
Generalization and Zero-Shot Transfer
By learning the "language" of motion rather than specific rules, these models exhibit remarkable generalization. Sixten Norelius showed with SketchPlan that a diffusion model can translate a human's 2D hand-drawn sketch into a 3D drone flight path in a cluttered environment it has never seen before.
In robotics, this manifests as "sim-to-real" success. Models like NavDP and VidBot (by Hanzhi Chen) demonstrate that by training on diverse generative distributions—including those distilled from human videos—robots can perform complex manipulation tasks in the real world without ever seeing real-world training data. The generative process acts as a bridge, capturing the essential multi-modal nature of movement that transcends specific sensors or environments.