Robot Motion Planning as Video Prediction: A Spatio-Temporal Neural Network-based Motion Planner

Xiao Zang, Miao Yin, Lingyi Huang, Jingjin Yu, Saman A. Zonouz, Bo Yuan

Abstract

Neural network (NN)-based methods have emerged as an attractive approach for robot motion planning due to strong learning capabilities of NN models and their inherently high parallelism. Despite the current development in this direction, the efficient capture and processing of important sequential and spatial information, in a direct and simultaneous way, is still relatively under-explored. To overcome the challenge and unlock the potentials of neural networks for motion planning tasks, in this paper, we propose STP-Net, an end-to-end learning framework that can fully extract and leverage important spatio-temporal information to form an efficient neural motion planner. By interpreting the movement of the robot as a video clip, robot motion planning is transformed to a video prediction task that can be performed by STP-Net in both spatially and temporally efficient ways. Empirical evaluations across different seen and unseen environments show that, with nearly 100% accuracy (aka, success rate), STP-Net demonstrates very promising performance with respect to both planning speed and path cost. Compared with existing NN-based motion planners, STP-Net achieves at least 5x, 2.6x and 1.8x faster speed with lower path cost on 2D Random Forest, 2D Maze and 3D Random Forest environments, respectively. Furthermore, STP-Net can quickly and simultaneously compute multiple near-optimal paths in multi-robot motion planning tasks.

Introduction

Finding a collision-free path from a starting point to a goal is a fundamental challenge in robotics. Traditional Sampling-based Motion Planners (SMPs), such as RRT*, are reliable but often computationally expensive because they perform extensive collision checks in areas far from the optimal path. Recent neural network (NN) approaches have attempted to accelerate this process by learning "biased sampling" to guide the search. However, many existing models suffer from two primary limitations: temporal blindness and spatial information loss. Standard MLPs or CNNs are often not designed to handle the sequential nature of a path, and the use of Auto-Encoders to compress environments into small vectors frequently destroys fine-grained spatial relationships between the robot and obstacles.

Methodology: STP-Net

STP-Net reimagines robot motion planning as a video prediction task. By treating the movement of a robot as a sequence of video frames, the model can "foresee" the path a robot should take.

Motion as Video

In this framework, each "frame" in the sequence contains three distinct channels: the static obstacle map, the robot’s current position, and the goal position. This representation allows the model to maintain a constant awareness of the environment while tracking progress toward the objective.

Spatio-Temporal Architecture

The core of STP-Net is a backbone consisting of four layers of Convolutional Long Short-Term Memory (ConvLSTM). This architecture is critical as it allows the model to process 2D or 3D spatial data while simultaneously capturing the temporal "flow" of the path. Unlike previous models that rely on compressed vector embeddings, STP-Net works directly on raw grid maps, preserving the spatial integrity of the environment.

Implementation Details

To improve training stability, the authors utilize "pixel patching," representing the robot and goal as small $p \times p$ patches rather than single pixels. The model outputs a probability map for the next waypoint; the most probable pixels are typically collision-free, which significantly reduces the need for exhaustive collision checking. Furthermore, the model can be extended to multi-robot planning by adding additional channels to the input, allowing for simultaneous path generation for multiple agents.

Experimental Evaluation

The model was evaluated across three distinct environments to test its robustness and generalization:

  1. 2D Random Forest: $64 \times 64$ grids with 15 randomly placed obstacles.
  2. 2D Maze: $64 \times 64$ grids with complex, narrow passages generated via randomized depth-first search.
  3. 3D Random Forest: $20 \times 20 \times 20$ voxel grids with 8 random obstacle blocks.

Ground truth data for training was generated using the RRT* algorithm to provide near-optimal paths for the network to emulate.

Results and Discussion

STP-Net demonstrated superior performance compared to both classical planners (Informed-RRT, BIT) and state-of-the-art neural planners (MPNet, OracleNet):

  • Success Rate: The model achieved nearly 100% accuracy in both seen and unseen environments.
  • Computational Speed: STP-Net is significantly faster than existing NN-based planners, showing a 5x speedup in 2D Random Forests, 2.6x in 2D Mazes, and 1.8x in 3D environments.
  • Efficiency: The model requires at least 22x fewer waypoint predictions and collision checks than existing learning-based solutions.
  • Path Quality: The resulting paths were shorter and lower in cost than those produced by competing neural planners, even outperforming BIT* under equivalent time constraints.

Conclusion

STP-Net demonstrates that treating motion planning as a spatio-temporal prediction task can unlock significant gains in speed and reliability. By avoiding the information loss of vector compression and leveraging the sequential processing power of ConvLSTMs, the framework provides a highly efficient alternative to traditional and existing neural planners. Future work will focus on integrating neural collision checkers directly into the pipeline, scaling to high-degree-of-freedom robotic arms, and adapting the framework for dynamic environments with moving obstacles.

Summary updated

Abstract

Neural network (NN)-based methods have emerged as an attractive approach for robot motion planning due to strong learning capabilities of NN models and their inherently high parallelism. Despite the current development in this direction, the efficient capture and processing of important sequential and spatial information, in a direct and simultaneous way, is still relatively under-explored. To overcome the challenge and unlock the potentials of neural networks for motion planning tasks, in this paper, we propose STP-Net, an end-to-end learning framework that can fully extract and leverage important spatio-temporal information to form an efficient neural motion planner. By interpreting the movement of the robot as a video clip, robot motion planning is transformed to a video prediction task that can be performed by STP-Net in both spatially and temporally efficient ways. Empirical evaluations across different seen and unseen environments show that, with nearly 100% accuracy (aka, success rate), STP-Net demonstrates very promising performance with respect to both planning speed and path cost. Compared with existing NN-based motion planners, STP-Net achieves at least 5x, 2.6x and 1.8x faster speed with lower path cost on 2D Random Forest, 2D Maze and 3D Random Forest environments, respectively. Furthermore, STP-Net can quickly and simultaneously compute multiple near-optimal paths in multi-robot motion planning tasks
BibTeX
@inproceedings{zang2022robot,
  author = {Xiao Zang and Miao Yin and Lingyi Huang and Jingjin Yu and Saman A. Zonouz and Bo Yuan},
  title = {{Robot Motion Planning as Video Prediction: A Spatio-Temporal Neural Network-based Motion Planner}},
  booktitle = {IEEE/RSJ International Conference on Intelligent Robots and Systems},
  year = {2022},
  doi = {10.1109/iros47612.2022.9981769},
}
Figure