Information-Theoretic Graph Fusion with Vision-Language-Action Model for Policy Reasoning and Dual Robotic Control

Shunlei Li, Longsen Gao, Jin Wang, Chang Che, Xi Xiao, Jiuwen Cao, Yingbai Hu, Hamid Reza Karimi

Abstract

Teaching robots dexterous skills from human videos remains challenging due to the reliance on low-level trajectory imitation, which fails to generalize across object types, spatial layouts, and manipulator configurations. We propose Graph-Fused Vision-Language-Action (GF-VLA), a framework that enables dual-arm robotic systems to perform task-level reasoning and execution directly from RGB and Depth human demonstrations. GF-VLA first extracts Shannon-information-based cues to identify hands and objects with the highest task relevance, then encodes these cues into temporally ordered scene graphs that capture both hand-object and object-object interactions. These graphs are fused with a language-conditioned transformer that generates hierarchical behavior trees and interpretable Cartesian motion commands. To improve execution efficiency in bimanual settings, we further introduce a cross-hand selection policy that infers optimal gripper assignment without explicit geometric reasoning. We evaluate GF-VLA on four structured dual-arm block assembly tasks involving symbolic shape construction and spatial generalization. Experimental results show that the information-theoretic scene representation achieves over 95 percent graph accuracy and 93 percent subtask segmentation, supporting the LLM planner in generating reliable and human-readable task policies. When executed by the dual-arm robot, these policies yield 94 percent grasp success, 89 percent placement accuracy, and 90 percent overall task success across stacking, letter-building, and geometric reconfiguration scenarios, demonstrating strong generalization and robustness across diverse spatial and semantic variations.

Introduction

Teaching robots complex, dexterous skills—such as building structures from blocks—traditionally faces a trade-off between brittle low-level trajectory imitation and high-level Vision-Language-Action (VLA) models that lack physical "common sense." Current VLA models often struggle with fine-grained physical interactions, efficient bimanual coordination, and grounding ambiguous language in scene geometry.

The GF-VLA framework translates high-level instructions into bimanual robotic actions through a vision-language-action loop.

The GF-VLA (Graph-Fused Vision-Language-Action) framework addresses these challenges by bridging high-level semantic reasoning with low-level physical control. By utilizing Shannon information theory to extract structured scene graphs from human demonstrations, the system enables a dual-arm robot to perform Chain-of-Thought (CoT) planning and execute precise motor commands.

Methods

The GF-VLA pipeline consists of three primary components: information-theoretic perception, a dual-head transformer architecture, and a bimanual coordination policy.

Information-Theoretic Scene Graphs

Rather than relying solely on pixel-level tracking, GF-VLA calculates Shannon entropy and Mutual Information (MI) between hand and object trajectories. High MI indicates "Coupled-Motion" (a grasp), while a decrease in distance entropy signifies a "Docked" state (a placement). This creates a symbolic scene graph capturing the human's intent.

The perception pipeline extracts keyframes and scene graphs from human demonstrations using information-theoretic metrics.

Unified Dual-Head Architecture

The model utilizes a LLaMA 2 backbone with two specialized heads:

  1. LLM Head: Performs high-level planning via Chain-of-Thought (CoT), breaking tasks into subgoals and generating self-verification criteria (e.g., "Check if grippers are closed").
  2. Action Head: Generates low-level Cartesian motion commands at 5Hz for the robot arms.

Cross-Hand Selection Policy

To optimize bimanual execution, a policy assigns tasks to the "contralateral" (opposite) hand relative to the target location. This minimizes path length and prevents collisions without intensive geometric computation.

Experimental Results

The framework was evaluated using a dataset of 250 human demonstration videos and 240 robotic trials on a dual-arm platform (Universal Robots UR5e and UR10e).

  • Perception and Segmentation: The system achieved over 95% graph accuracy and 93% subtask segmentation from human videos.
  • Robotic Execution: In real-world tests, GF-VLA reached a 94% grasp success rate and a 90% overall task success rate across stacking and letter-building tasks.
  • Generalization: The model demonstrated an 86% plan transferability rate, successfully applying policies from one assembly task to novel ones.
  • Ablation Study: Removing the information-theoretic scene graph led to a 0.49 drop in Task Success Rate (TSR), highlighting the importance of structured physical priors.

Conclusion

GF-VLA demonstrates that combining information-theoretic perception with LLM-based reasoning significantly improves a robot's ability to learn from human demonstrations. While currently limited by calibration dependencies and rigid object assumptions, future work aims to incorporate temporal memory and address deformable objects to further enhance robotic dexterity and task-level autonomy.

Summary updated

Abstract

Teaching robots dexterous skills from human videos remains challenging due to the reliance on low-level trajectory imitation, which fails to generalize across object types, spatial layouts, and manipulator configurations. We propose Graph-Fused Vision-Language-Action (GF-VLA), a framework that enables dual-arm robotic systems to perform task-level reasoning and execution directly from RGB and Depth human demonstrations. GF-VLA first extracts Shannon-information-based cues to identify hands and objects with the highest task relevance, then encodes these cues into temporally ordered scene graphs that capture both hand-object and object-object interactions. These graphs are fused with a language-conditioned transformer that generates hierarchical behavior trees and interpretable Cartesian motion commands. To improve execution efficiency in bimanual settings, we further introduce a cross-hand selection policy that infers optimal gripper assignment without explicit geometric reasoning. We evaluate GF-VLA on four structured dual-arm block assembly tasks involving symbolic shape construction and spatial generalization. Experimental results show that the information-theoretic scene representation achieves over 95 percent graph accuracy and 93 percent subtask segmentation, supporting the LLM planner in generating reliable and human-readable task policies. When executed by the dual-arm robot, these policies yield 94 percent grasp success, 89 percent placement accuracy, and 90 percent overall task success across stacking, letter-building, and geometric reconfiguration scenarios, demonstrating strong generalization and robustness across diverse spatial and semantic variations.
BibTeX
@misc{li2025information,
  author = {Shunlei Li and Longsen Gao and Jin Wang and Chang Che and Xi Xiao and Jiuwen Cao and Yingbai Hu and Hamid Reza Karimi},
  title = {{Information-Theoretic Graph Fusion with Vision-Language-Action Model for Policy Reasoning and Dual Robotic Control}},
  year = {2025},
  eprint = {2508.05342},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2508.05342},
}
Figure