GraF-VTA: Neuro-Symbolic Trajectory Generation from Vision and Temporal Logic via Graph-Based Fusion

Zezhi Liu, Zhiwei Zheng, Hanqian Luo, Deyun Qin, Shizhen Wu, Yongchun FangarXiv 2026 · arXiv

TL;DR

GraF-VTA is a neuro-symbolic framework that generates robot trajectories by fusing visual observations with Signal Temporal Logic (STL) specifications. By combining structured task graphs, a "temporal pyramid" of visual history, and closed-loop replanning via flow-matching, it achieves significantly higher task success rates and planning efficiency than previous geometry-dependent or end-to-end planners.

What Problem Does This Paper Address?

Robots operating in the real world need to follow complex, time-sensitive instructions—like "pick up the red block within 10 seconds, then move it to the blue zone while avoiding the obstacle." Signal Temporal Logic (STL) is a powerful mathematical language for defining these requirements.

However, two major hurdles remain:

  1. Visual Grounding: Most STL planners require exact 3D coordinates of objects, which are difficult to extract reliably from raw camera pixels.
  2. Execution Robustness: Generative models often plan "open-loop," meaning they don't account for real-world disturbances or small errors that accumulate during movement.

GraF-VTA tackles these by learning to "ground" logical symbols directly into image features and using a receding-horizon approach to constantly refresh its plan based on what the robot actually sees.

Methods

The GraF-VTA architecture (Figure 1) is built on several novel components:

  • Graph-Based Fusion: The system represents STL commands as a syntax tree. A Graph Neural Network (GNN) encodes this tree into "logic tokens." Optionally, it can also incorporate a Deterministic Finite Automaton (DFA) embedding to capture the high-level, untimed structure of the task.
  • Temporal Pyramid Memory: To give the robot "context" without overwhelming it with data, the authors use a temporal pyramid. It keeps the current camera frame at full resolution but spatially compresses older frames. This allows the model to remember recent actions (like "I already picked up the block") while focusing on the immediate details needed for control.
  • Bidirectional Cross-Attention: This is the "secret sauce" that connects vision and logic. Logic tokens attend to visual features to find relevant objects, and visual tokens attend to logic tokens to understand which parts of the scene matter for the current task.
  • Flow-Matching Generator: Instead of standard diffusion, the model uses a flow-matching U-Net to generate trajectories. This approach is often faster and more stable for generating continuous robot paths.
  • Closed-Loop Replanning: Rather than executing a long plan to the end, GraF-VTA executes only a short "action prefix," then looks at the world again and generates a fresh plan.
Figure 1. System diagram for GraF-VTA. The STL specification is represented as a syntax tree and a deterministic finite automaton (DFA), both encoded by GNNs. Logic tokens from the syntax tree and visual tokens from multi-view observations are fused through bidirectional cross-attention. The fused features, DFA embedding, and current state condition a Temporal U-Net to generate the planned trajectory via an ODE solver.

Datasets

The authors evaluated GraF-VTA across five diverse simulation environments:

  • Linear & Dubins (ZoneEnv): Navigation tasks with simple and car-like dynamics.
  • PointMaze & AntMaze: Navigating complex mazes with a point mass or a quadruped "Ant" robot.
  • Franka Panda: A 7-DOF robotic arm performing manipulation tasks.

The evaluation used 128 distinct STL specification tasks per environment, covering single-goal, multi-goal, sequential, and partial-order requirements.

Key Insights and Ideas

  • History is Crucial for Logic: The "pyramid memory" wasn't just a minor tweak; on the Panda arm, it boosted success rates from ~54% to 62.5%. Knowing the history of observations helps the robot understand which stage of a multi-step STL task it is currently in.
  • Symbolic Attention: The model actually learns to associate words with pixels. As shown in Figure 4, when the researchers swapped object IDs in the STL command, the model’s visual attention shifted to the newly referenced objects, proving it wasn't just memorizing paths but actually "reading" the scene.
  • Efficiency Gains: By using flow-matching and structured embeddings, GraF-VTA achieved over a 5x speedup in planning latency compared to other closed-loop baselines like S-MSP and Logic-VLA.
Visual attention shifts when object IDs are swapped, demonstrating successful grounding.

Key Results

  • Open-Loop Superiority: In open-loop tests, GraF-VTA outperformed the previous state-of-the-art (TeLoGraF and VTA) in 4 out of 5 environments (Figure 3).
  • Closed-Loop Performance: On the challenging Dubins car environment, GraF-VTA reached a 64.06% success rate, dwarfing the 12.50% and 27.34% achieved by adapted versions of S-MSP and Logic-VLA, respectively.
  • Robustness to Complexity: The radar plots in Figure 5 show that closed-loop execution provides the biggest gains in "Sequential" and "Partial-Order" tasks, where the robot must manage multiple stages of a task over time.
Figure 3. Open-loop trajectory coverage. GraF-VTA, using only the STL syntax graph, outperforms TeLoGraF and VTA in Success@K on Linear, Dubins, PointMaze, and Panda at every plotted budget K ≥16.
Figure 5. STL satisfaction rates of open-loop and closed-loop GraF-VTA with dynamics execution across five environments and four task categories at K = 1024 (I: single-goal; II: multi-goal; III: sequential; IV: partial-order).

Limitations and Future Work

While GraF-VTA is a significant step forward, the authors noted a few areas for improvement:

  • AntMaze Performance: The model performed slightly worse than the VTA baseline in the AntMaze environment, suggesting that complex quadruped dynamics might require more specialized handling.
  • Static Automata: The current DFA embedding is "frozen" and doesn't explicitly track task progress in real-time; it only provides a static structural hint.
  • Sim-to-Real: All experiments were conducted in simulation. Future work will need to address the "reality gap" when moving these neuro-symbolic models to physical hardware with noisy sensors.

Summary updated

Abstract

Robotic planning from visual observations and signal temporal logic (STL) requires grounding task requirements in scene evidence while adapting plans to execution feedback. We present GraF-VTA, a graph-based framework that extends Vision-TL-Action with structured task representations, causal observation history, and closed-loop trajectory generation. Bidirectional cross-attention fuses STL syntax tokens with visual features, while an optional frozen automaton embedding supplies complementary untimed task structure. A temporal pyramid preserves detailed current observations and compresses older frames to provide recent execution context. Conditioned on these features and robot state, a flow-matching model generates plans whose action prefixes are executed before replanning from refreshed observations. Across five simulation environments, the single-graph configuration achieves higher open-loop Success@$K$ than TeLoGraF and Vision-TL-Action on four environments at the reported budgets $K\geq16$. On Panda, matched-update ablations increase Success@256 from 53.91\% without observation history to 62.50\% with pyramid memory. In closed-loop Dubins evaluation, it reaches 64.06\% Success@1024, compared with 12.50\% for adapted S-MSP and 27.34\% for adapted Logic-VLA. These results support combining structured visual conditioning with observation history and replanning to improve coverage of STL-satisfying trajectories.
BibTeX
@misc{liu2026graf,
  author = {Zezhi Liu and Zhiwei Zheng and Hanqian Luo and Deyun Qin and Shizhen Wu and Yongchun Fang},
  title = {{GraF-VTA: Neuro-Symbolic Trajectory Generation from Vision and Temporal Logic via Graph-Based Fusion}},
  year = {2026},
  eprint = {2611.25589},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.25589},
}
Figure