Bridging Perception and Planning: Towards End-to-End Planning for Signal Temporal Logic Tasks

Xiang Yin (殷翔), Bowen Ye, Junyue Huang, Yang Liu, Xiaozhen Qiao, Xiang YinarXiv 2025 · DOI · arXiv

Robotic systems operating in the real world must often navigate complex environments while adhering to time-sensitive and spatially constrained instructions. Traditionally, this requires a multi-stage pipeline: perceiving the environment, building a map, localizing the robot, and then planning a path. However, these pipelines are brittle; a single error in perception can lead to a total failure in planning. This paper introduces S-MSP, a framework designed to bridge the gap between raw perception and high-level planning through an end-to-end approach.

The Challenge of End-to-End STL Planning

Signal Temporal Logic (STL) provides a formal way to describe complex tasks, such as "Visit the charging station within 10 seconds, then stay in the loading zone for 5 seconds, while always avoiding obstacles." Planning for these tasks typically assumes a structured world with pre-defined maps and known dynamics. In unstructured, real-world environments where only raw camera feeds are available, the challenge is to map pixels and logic formulas directly to safe, feasible trajectories without intermediate mapping steps.

Methodology: The S-MSP Framework

The core contribution of this work is the Structured MoE STL Planner (S-MSP). The architecture is designed to handle the compositional nature of logic and the high dimensionality of visual data through several key innovations.

Multi-Source Encoding

S-MSP processes four distinct data streams to inform its planning:

  1. Visual Data: Multi-view RGB images processed via a Swin-B backbone.
  2. Task Specification: The STL formula provided as text.
  3. State Information: The robot's current starting pose.
  4. Structural Encoding: A specialized representation of the STL formula's logical hierarchy.

Structure-Aware Mixture-of-Experts (MoE)

Instead of a standard dense transformer, S-MSP utilizes a structure-aware MoE architecture. A router analyzes the STL formula and assigns specific "experts" to handle different logical operators (e.g., "Always", "Eventually") and specific time intervals. This temporal anchoring allows the model to manage long-horizon dependencies more effectively by specializing experts for different phases of the task.

The S-MSP architecture featuring the structure-aware Mixture-of-Experts (MoE) module.

Differentiable Robustness and Safety

To train the network, the authors employ a "smooth" version of STL robustness, allowing the satisfaction of logical constraints to serve as a differentiable loss function. Furthermore, a Triggered Segment Repair (TSP) layer is integrated as a safety mechanism. If the neural network's predicted trajectory is flagged for potential collisions or logic violations, a rule-based RRT planner intervenes to repair the problematic segment.

Experimental Evaluation

The researchers developed a high-fidelity benchmark using the Gazebo simulator, modeling a factory logistics environment with varying obstacles like aisles, racks, and pallets. The model was tested on both In-Distribution (ID) layouts and Out-of-Distribution (OOD) scenarios with entirely unseen warehouse configurations.

A sample trajectory generated by S-MSP in the warehouse environment.

Key Results

S-MSP demonstrated significant performance gains over standard transformer baselines:

  • Success Rate: In ID tasks, S-MSP achieved a 71.00% success rate, outperforming the baseline's 61.50%.
  • Safety Integration: With the addition of the TSP repair layer, the success rate rose to 88.00%, a substantial improvement over the baseline combined with TSP.
  • Generalization: The model maintained its performance lead in OOD scenarios, indicating that it learned to generalize the relationship between visual cues and logical constraints rather than simply memorizing specific maps.

Conclusion and Future Work

S-MSP represents a significant step toward autonomous robots that can interpret complex instructions directly from visual perception. By using logic to guide neural routing, the framework achieves better generalization and task success. Future work aims to extend this approach to more complex, nested logical operators and to incorporate reinforcement learning to better align robot behavior with human preferences.

Summary updated

Abstract

We investigate the task and motion planning problem for Signal Temporal Logic (STL) specifications in robotics. Existing STL methods rely on pre-defined maps or mobility representations, which are ineffective in unstructured real-world environments. We propose the \emph{Structured-MoE STL Planner} (\textbf{S-MSP}), a differentiable framework that maps synchronized multi-view camera observations and an STL specification directly to a feasible trajectory. S-MSP integrates STL constraints within a unified pipeline, trained with a composite loss that combines trajectory reconstruction and STL robustness. A \emph{structure-aware} Mixture-of-Experts (MoE) model enables horizon-aware specialization by projecting sub-tasks into temporally anchored embeddings. We evaluate S-MSP using a high-fidelity simulation of factory-logistics scenarios with temporally constrained tasks. Experiments show that S-MSP outperforms single-expert baselines in STL satisfaction and trajectory feasibility. A rule-based \emph{safety filter} at inference improves physical executability without compromising logical correctness, showcasing the practicality of the approach.
BibTeX
@misc{2025bridging,
  author = {Xiang Yin (殷翔) and Bowen Ye and Junyue Huang and Yang Liu and Xiaozhen Qiao and Xiang Yin},
  title = {{Bridging Perception and Planning: Towards End-to-End Planning for Signal Temporal Logic Tasks}},
  year = {2025},
  eprint = {2509.12813},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2509.12813},
}
Figure