Robotic systems operating in the real world must often navigate complex environments while adhering to time-sensitive and spatially constrained instructions. Traditionally, this requires a multi-stage pipeline: perceiving the environment, building a map, localizing the robot, and then planning a path. However, these pipelines are brittle; a single error in perception can lead to a total failure in planning. This paper introduces S-MSP, a framework designed to bridge the gap between raw perception and high-level planning through an end-to-end approach.
The Challenge of End-to-End STL Planning
Signal Temporal Logic (STL) provides a formal way to describe complex tasks, such as "Visit the charging station within 10 seconds, then stay in the loading zone for 5 seconds, while always avoiding obstacles." Planning for these tasks typically assumes a structured world with pre-defined maps and known dynamics. In unstructured, real-world environments where only raw camera feeds are available, the challenge is to map pixels and logic formulas directly to safe, feasible trajectories without intermediate mapping steps.
Methodology: The S-MSP Framework
The core contribution of this work is the Structured MoE STL Planner (S-MSP). The architecture is designed to handle the compositional nature of logic and the high dimensionality of visual data through several key innovations.
Multi-Source Encoding
S-MSP processes four distinct data streams to inform its planning:
- Visual Data: Multi-view RGB images processed via a Swin-B backbone.
- Task Specification: The STL formula provided as text.
- State Information: The robot's current starting pose.
- Structural Encoding: A specialized representation of the STL formula's logical hierarchy.
Structure-Aware Mixture-of-Experts (MoE)
Instead of a standard dense transformer, S-MSP utilizes a structure-aware MoE architecture. A router analyzes the STL formula and assigns specific "experts" to handle different logical operators (e.g., "Always", "Eventually") and specific time intervals. This temporal anchoring allows the model to manage long-horizon dependencies more effectively by specializing experts for different phases of the task.
Differentiable Robustness and Safety
To train the network, the authors employ a "smooth" version of STL robustness, allowing the satisfaction of logical constraints to serve as a differentiable loss function. Furthermore, a Triggered Segment Repair (TSP) layer is integrated as a safety mechanism. If the neural network's predicted trajectory is flagged for potential collisions or logic violations, a rule-based RRT planner intervenes to repair the problematic segment.
Experimental Evaluation
The researchers developed a high-fidelity benchmark using the Gazebo simulator, modeling a factory logistics environment with varying obstacles like aisles, racks, and pallets. The model was tested on both In-Distribution (ID) layouts and Out-of-Distribution (OOD) scenarios with entirely unseen warehouse configurations.
Key Results
S-MSP demonstrated significant performance gains over standard transformer baselines:
- Success Rate: In ID tasks, S-MSP achieved a 71.00% success rate, outperforming the baseline's 61.50%.
- Safety Integration: With the addition of the TSP repair layer, the success rate rose to 88.00%, a substantial improvement over the baseline combined with TSP.
- Generalization: The model maintained its performance lead in OOD scenarios, indicating that it learned to generalize the relationship between visual cues and logical constraints rather than simply memorizing specific maps.
Conclusion and Future Work
S-MSP represents a significant step toward autonomous robots that can interpret complex instructions directly from visual perception. By using logic to guide neural routing, the framework achieves better generalization and task success. Future work aims to extend this approach to more complex, nested logical operators and to incorporate reinforcement learning to better align robot behavior with human preferences.
Summary updated