Robot Motion Planning as Video Prediction: A Spatio-Temporal Neural Network-based Motion Planner

Xiao Zang, Miao Yin, Lingyi Huang, Jingjin Yu, Saman A. Zonouz, Bo Yuan

TL;DR

STP-Net reimagines robot motion planning as a video prediction task, using a spatio-temporal neural network to "foresee" the path a robot should take. By leveraging ConvLSTM layers to process raw environment maps directly, it achieves nearly 100% success rates while being up to 5x faster than existing neural planners.

What Problem Does This Paper Address?

Finding a collision-free path from point A to point B is a fundamental challenge in robotics. Traditional Sampling-based Motion Planners (SMPs) like RRT* are reliable but slow, often wasting time checking collisions in areas far from the optimal path.

Recent neural network (NN) approaches try to speed this up by learning "biased sampling" (guessing where the path should be). However, these existing models have two major flaws:

  1. Temporal Blindness: They often use standard MLPs or CNNs, which aren't designed to handle the inherently sequential nature of a path (where the next step depends heavily on the previous ones).
  2. Spatial Information Loss: Many models use "Auto-Encoders" to compress the environment into a small vector. This compression often destroys the fine-grained spatial relationships between the robot and nearby obstacles.

Methods

The authors introduce STP-Net, which treats the motion of a robot as a sequence of video frames.

  • Motion as Video: Each "frame" in the video contains three channels: the static obstacle map, the robot’s current position, and the goal position.
  • ConvLSTM Backbone: Instead of standard NNs, STP-Net uses four layers of Convolutional Long Short-Term Memory (ConvLSTM). This architecture is the "secret sauce"—it allows the model to process 2D/3D spatial data while simultaneously remembering the temporal "flow" of the path.
  • Direct Processing: Unlike previous models, STP-Net works directly on the raw grid maps. This avoids the information loss associated with vector embeddings.
  • Pixel Patching: To help the network "see" the robot and goal better, the authors represent them as small $p \times p$ patches rather than single pixels, which improves training stability and performance.
  • Multi-Robot Extension: The model can be scaled to multi-robot planning by simply adding more channels to the input "video," allowing robots to plan paths simultaneously while avoiding one another.

Datasets

The model was trained and evaluated on three distinct environments:

  1. 2D Random Forest: $64 \times 64$ grids with 15 randomly placed obstacles of varying shapes.
  2. 2D Maze: $64 \times 64$ grids generated via randomized depth-first search, creating narrow, complex passages.
  3. 3D Random Forest: $20 \times 20 \times 20$ voxel grids with 8 random obstacle blocks.

Ground truth data for training was generated using the RRT* algorithm to provide near-optimal paths.

Key Insights and Ideas

  • The "Aha" Moment: Motion planning is essentially predicting the next frame in a video. If a model can predict what the "scene" looks like one step into the future, it has effectively found the next waypoint.
  • Confidence is Key: STP-Net outputs a probability map for the next waypoint. The authors found that the "brightest" (most probable) pixels are almost always collision-free, which drastically reduces the number of expensive collision checks needed during the planning phase.
  • Efficiency through Structure: By using ConvLSTMs, the network naturally learns the spatial correlation between the robot and obstacles. It doesn't just know where the robot is; it understands the robot's position relative to the walls in a way that compressed vectors cannot.

Key Results

STP-Net outperformed both classical planners (Informed-RRT, BIT) and state-of-the-art neural planners (MPNet, OracleNet) across the board:

  • Success Rate: Achieved nearly 100% accuracy in both seen and unseen environments.
  • Speed: It is 5x faster in 2D Random Forests, 2.6x faster in 2D Mazes, and 1.8x faster in 3D environments compared to other NN-based planners.
  • Reduced Computation: STP-Net requires at least 22x fewer waypoint predictions and collision checks than existing learning-based solutions to find a valid path.
  • Path Quality: The paths produced were shorter (lower cost) than those from other neural planners and even outperformed BIT* when given the same time budget.

Limitations and Future Work

While STP-Net is highly efficient, the authors note a few areas for growth:

  • Collision Checking: Currently, the model still relies on a standard external collision detection function to verify the final path. The authors plan to integrate a "neural collision checker" directly into the pipeline.
  • High-Dimensional Scaling: The current work focuses on 2D and 3D workspaces. Extending this to high-degree-of-freedom robots (like 7-axis robotic arms) where the "map" is much more complex remains a future challenge.
  • Dynamic Obstacles: The current study focuses on static environments; adapting the video prediction framework for moving obstacles is a logical next step.

Summary updated

Abstract

Neural network (NN)-based methods have emerged as an attractive approach for robot motion planning due to strong learning capabilities of NN models and their inherently high parallelism. Despite the current development in this direction, the efficient capture and processing of important sequential and spatial information, in a direct and simultaneous way, is still relatively under-explored. To overcome the challenge and unlock the potentials of neural networks for motion planning tasks, in this paper, we propose STP-Net, an end-to-end learning framework that can fully extract and leverage important spatio-temporal information to form an efficient neural motion planner. By interpreting the movement of the robot as a video clip, robot motion planning is transformed to a video prediction task that can be performed by STP-Net in both spatially and temporally efficient ways. Empirical evaluations across different seen and unseen environments show that, with nearly 100% accuracy (aka, success rate), STP-Net demonstrates very promising performance with respect to both planning speed and path cost. Compared with existing NN-based motion planners, STP-Net achieves at least 5x, 2.6x and 1.8x faster speed with lower path cost on 2D Random Forest, 2D Maze and 3D Random Forest environments, respectively. Furthermore, STP-Net can quickly and simultaneously compute multiple near-optimal paths in multi-robot motion planning tasks
BibTeX
@inproceedings{zang2022robot,
  author = {Xiao Zang and Miao Yin and Lingyi Huang and Jingjin Yu and Saman A. Zonouz and Bo Yuan},
  title = {{Robot Motion Planning as Video Prediction: A Spatio-Temporal Neural Network-based Motion Planner}},
  booktitle = {IEEE/RSJ International Conference on Intelligent Robots and Systems},
  year = {2022},
  doi = {10.1109/iros47612.2022.9981769},
}

Loading related content...

Figure