PathFormer: A Transformer with 3D Grid Constraints for Digital Twin Robot-Arm Trajectory Generation

Ahmed Alanazi, Duy Ho, Yugyung LeearXiv 2025 · DOI · arXiv

Introduction

The integration of Transformers into robotics has significantly advanced language-conditioned manipulation. However, these high-level sequence models often operate in a vacuum, detached from the low-level physical realities of the robot's workspace. Standard autoregressive generation can produce "brittle" trajectories that are kinematically impossible, discontinuous, or out of bounds. PathFormer addresses this fundamental gap by treating the workspace as a constrained 3D grid and embedding geometric and kinematic constraints directly into the Transformer’s decoding process. By ensuring "legality by construction," PathFormer bridges the divide between abstract planning and physical execution.

Methodology: Graph-Constrained Decoding

The core of the PathFormer architecture is a 3-grid representation that unifies three critical dimensions of any robotic task: spatial, semantic, and temporal.

  1. Spatial: The workspace is discretized into a 3D lattice with a 20mm resolution, providing a high-fidelity map for movement.
  2. Semantic: Tasks are decomposed into a graph of primitives, such as "grasp," "lift," and "place."
  3. Temporal: The model maintains the sequential order necessary for complex multi-step execution.
PathFormer system architecture: from depth sensing to 3D grid path generation and physical execution.

The standout technical contribution is Graph-Constrained Decoding. Unlike traditional models that might pick any coordinate in the workspace, PathFormer applies a neighborhood mask ($M_t$) during generation. This mask restricts the search space to only those cells that are lattice-adjacent (a Manhattan distance of 1) and within the valid workspace bounds. This ensures that the robot never attempts a "teleportation" move or an out-of-bounds action.

This model is integrated into a Digital Twin pipeline. Utilizing a RealSense depth camera and a YOLOv11 detector, the system projects real-world objects into a voxelized 3D grid. This grid synchronizes the "digital twin" simulation with the physical xArm Lite 6, allowing for real-time safety checks and local re-grounding.

Experimental Evaluation

PathFormer was rigorously trained and evaluated across three primary datasets to ensure both theoretical fidelity and real-world robustness.

  • Lattice Corpus: A foundational dataset of 53,755 trajectories used to teach the model coordinate recovery and trajectory fidelity.
  • Controlled Benchmarks (D2): Specific tests designed to evaluate "reach" and "pick" primitives across various workspace ranges.
  • Real-World Cluttered Scenes (D3): 60 natural-language instructions (e.g., "pick up smartphone > place in container") performed in dynamic environments featuring partial occlusions and moving obstacles.
Digital twin synchronization: real-time object detection and depth mapping integrated with the 3D grid trajectory planner.

Key Results and Insights

The implementation of neighborhood masking resulted in a 99.99% valid path rate, effectively eliminating illegal moves at the generation stage. In offline evaluations, the model achieved 89.44% stepwise accuracy and 93.32% precision in recovering ground-truth trajectories.

Physical execution metrics were equally impressive. In controlled environments, the system achieved a 97.5% success rate for reaching and 92.5% for picking. In the more challenging D3 benchmark—involving complex, language-specified tasks in cluttered environments—PathFormer maintained an 86.7% success rate.

A critical insight from these tests is the model's capacity for local re-grounding. Because the model is grid-based and autoregressive, it exhibits "organic" error recovery. If an object slips or an obstacle enters the path, the model can take a "one-cell detour" to rejoin its nominal route without needing to discard the entire plan and restart from scratch. This unified latent space—fusing geometry, semantics, and time—allows the Transformer to reason about why it is moving alongside where it is moving.

Conclusion and Future Work

PathFormer demonstrates that embedding physical constraints into the generative process of a Transformer can yield highly reliable and safe robot trajectories. While the system is robust, the authors identified several areas for future development. "Tail truncation" on long-horizon tasks remains a challenge, potentially requiring horizon-aware decoding. Additionally, perception bottlenecks caused by heavy occlusions accounted for the majority of failures (8.3%), suggesting a need for multi-view sensing. Future iterations will also look to couple PathFormer with higher-level LLM planners to handle more abstract reasoning while maintaining low-level safety guarantees.

Summary updated

Abstract

Robotic arms require precise, task-aware trajectory planning, yet sequence models that ignore motion structure often yield invalid or inefficient executions. We present a Path-based Transformer that encodes robot motion with a 3-grid (where/what/when) representation and constraint-masked decoding, enforcing lattice-adjacent moves and workspace bounds while reasoning over task graphs and action order. Trained on 53,755 trajectories (80% train / 20% validation), the model aligns closely with ground truth -- 89.44% stepwise accuracy, 93.32% precision, 89.44% recall, and 90.40% F1 -- with 99.99% of paths legal by construction. Compiled to motor primitives on an xArm Lite 6 with a depth-camera digital twin, it attains up to 97.5% reach and 92.5% pick success in controlled tests, and 86.7% end-to-end success across 60 language-specified tasks in cluttered scenes, absorbing slips and occlusions via local re-grounding without global re-planning. These results show that path-structured representations enable Transformers to generate accurate, reliable, and interpretable robot trajectories, bridging graph-based planning and sequence-based learning and providing a practical foundation for general-purpose manipulation and sim-to-real transfer.
BibTeX
@misc{alanazi2025pathformer,
  author = {Ahmed Alanazi and Duy Ho and Yugyung Lee},
  title = {{PathFormer: A Transformer with 3D Grid Constraints for Digital Twin Robot-Arm Trajectory Generation}},
  year = {2025},
  eprint = {2510.20161},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2510.20161},
}
Figure