TL;DR
GraF-VTA is a neuro-symbolic framework that generates robot trajectories by fusing visual observations with Signal Temporal Logic (STL) specifications. By combining structured task graphs, a "temporal pyramid" of visual history, and closed-loop replanning via flow-matching, it achieves significantly higher task success rates and planning efficiency than previous geometry-dependent or end-to-end planners.
What Problem Does This Paper Address?
Robots operating in the real world need to follow complex, time-sensitive instructions—like "pick up the red block within 10 seconds, then move it to the blue zone while avoiding the obstacle." Signal Temporal Logic (STL) is a powerful mathematical language for defining these requirements.
However, two major hurdles remain:
- Visual Grounding: Most STL planners require exact 3D coordinates of objects, which are difficult to extract reliably from raw camera pixels.
- Execution Robustness: Generative models often plan "open-loop," meaning they don't account for real-world disturbances or small errors that accumulate during movement.
GraF-VTA tackles these by learning to "ground" logical symbols directly into image features and using a receding-horizon approach to constantly refresh its plan based on what the robot actually sees.
Methods
The GraF-VTA architecture (Figure 1) is built on several novel components:
- Graph-Based Fusion: The system represents STL commands as a syntax tree. A Graph Neural Network (GNN) encodes this tree into "logic tokens." Optionally, it can also incorporate a Deterministic Finite Automaton (DFA) embedding to capture the high-level, untimed structure of the task.
- Temporal Pyramid Memory: To give the robot "context" without overwhelming it with data, the authors use a temporal pyramid. It keeps the current camera frame at full resolution but spatially compresses older frames. This allows the model to remember recent actions (like "I already picked up the block") while focusing on the immediate details needed for control.
- Bidirectional Cross-Attention: This is the "secret sauce" that connects vision and logic. Logic tokens attend to visual features to find relevant objects, and visual tokens attend to logic tokens to understand which parts of the scene matter for the current task.
- Flow-Matching Generator: Instead of standard diffusion, the model uses a flow-matching U-Net to generate trajectories. This approach is often faster and more stable for generating continuous robot paths.
- Closed-Loop Replanning: Rather than executing a long plan to the end, GraF-VTA executes only a short "action prefix," then looks at the world again and generates a fresh plan.
Datasets
The authors evaluated GraF-VTA across five diverse simulation environments:
- Linear & Dubins (ZoneEnv): Navigation tasks with simple and car-like dynamics.
- PointMaze & AntMaze: Navigating complex mazes with a point mass or a quadruped "Ant" robot.
- Franka Panda: A 7-DOF robotic arm performing manipulation tasks.
The evaluation used 128 distinct STL specification tasks per environment, covering single-goal, multi-goal, sequential, and partial-order requirements.
Key Insights and Ideas
- History is Crucial for Logic: The "pyramid memory" wasn't just a minor tweak; on the Panda arm, it boosted success rates from ~54% to 62.5%. Knowing the history of observations helps the robot understand which stage of a multi-step STL task it is currently in.
- Symbolic Attention: The model actually learns to associate words with pixels. As shown in Figure 4, when the researchers swapped object IDs in the STL command, the model’s visual attention shifted to the newly referenced objects, proving it wasn't just memorizing paths but actually "reading" the scene.
- Efficiency Gains: By using flow-matching and structured embeddings, GraF-VTA achieved over a 5x speedup in planning latency compared to other closed-loop baselines like S-MSP and Logic-VLA.
Key Results
- Open-Loop Superiority: In open-loop tests, GraF-VTA outperformed the previous state-of-the-art (TeLoGraF and VTA) in 4 out of 5 environments (Figure 3).
- Closed-Loop Performance: On the challenging Dubins car environment, GraF-VTA reached a 64.06% success rate, dwarfing the 12.50% and 27.34% achieved by adapted versions of S-MSP and Logic-VLA, respectively.
- Robustness to Complexity: The radar plots in Figure 5 show that closed-loop execution provides the biggest gains in "Sequential" and "Partial-Order" tasks, where the robot must manage multiple stages of a task over time.
Limitations and Future Work
While GraF-VTA is a significant step forward, the authors noted a few areas for improvement:
- AntMaze Performance: The model performed slightly worse than the VTA baseline in the AntMaze environment, suggesting that complex quadruped dynamics might require more specialized handling.
- Static Automata: The current DFA embedding is "frozen" and doesn't explicitly track task progress in real-time; it only provides a static structural hint.
- Sim-to-Real: All experiments were conducted in simulation. Future work will need to address the "reality gap" when moving these neuro-symbolic models to physical hardware with noisy sensors.
Summary updated