Generate Subgoal Images Before Act: Unlocking the Chain-of-Thought Reasoning in Diffusion Model for Robot Manipulation with Multimodal Prompts

Fei Ni, Jianye Hao, Shiguang Wu, Longxin Kou, Jiashun Liu, Yan Zheng, Bin Wang, Yuzheng Zhuang

Introduction

Robotic agents frequently struggle with long-horizon tasks that involve complex, multimodal instructions. For instance, a command like "Put the [image of a block] into the [image of a container] without crossing the [image of a line]" requires both spatial reasoning and temporal planning. Traditional end-to-end models often suffer from error accumulation, while LLM-based planners that decompose tasks into text-based steps often fail to capture the nuanced spatial constraints necessary for physical grounding.

Motivation for visual subgoals: text-based instructions can be overly complex and difficult for robots to ground compared to visual keyframes.

The core insight of this work is that "a picture is worth a thousand words." By generating a sequence of mental subgoal images—a visual "Chain-of-Thought" (CoT)—robots can decompose complex prompts into achievable visual milestones, significantly improving success rates in long-horizon manipulation tasks.

The CoTDiffusion Framework

CoTDiffusion is a hierarchical framework designed to bridge the gap between high-level planning and low-level execution. It consists of two primary components: a high-level visual planner and a low-level action policy.

Visual Chain-of-Thought (CoT)

Instead of predicting motor actions directly from a complex prompt, a conditional diffusion model generates a sequence of keyframes. These keyframes serve as intermediate visual goals, effectively breaking a long-horizon task into manageable segments. This approach allows the model to "think" in the visual domain before acting.

Semantic Alignment Module (SAM)

To ensure the diffusion model generates the correct next step, the Semantic Alignment Module (SAM) acts as a progress tracker. It utilizes a "triple alignment" architecture that compares the initial observation, the current state, and the multimodal prompt to determine the current stage of the task. This module ensures that the generated subgoals are tightly grounded in the original instructions.

Training and Consistency

The model is trained in two stages. First, the SAM undergoes "coarse alignment" via a masked patch prediction task to learn which parts of a scene change between subgoals. Second, the diffusion model is trained using bi-directional generation, requiring it to predict both the next subgoal and reconstruct the previous one. This ensures logical consistency and prevents the "hallucinations" common in standard generative models.

Experimental Evaluation

The researchers evaluated CoTDiffusion on the VIMA-BENCH, focusing on three challenging categories: Visual Rearrange (matching a specific setup), Visual Reasoning (logic-based instructions), and Visual Constraints (adhering to spatial boundaries).

Key Results

CoTDiffusion demonstrated superior performance compared to state-of-the-art baselines, including Gato, Flamingo, VIMA, and SuSIE.

Success rates of CoTDiffusion compared to baselines across different task difficulty levels (L1-L3).

As shown in the results, CoTDiffusion achieved an overall success rate of 64.6%, significantly outperforming VIMA (49.5%). Notably, it maintained strong performance even in "L3" tasks involving novel object-task combinations, where other models struggled. Comparisons with SuSIE further highlighted that generated images are more effective for robot planning than human-written text sub-prompts, as SuSIE's performance lagged even when provided with manual text decompositions.

Conclusion

By treating image generation as a visual Chain-of-Thought, CoTDiffusion allows robots to plan effectively in the visual domain. This decoupling of high-level planning from low-level execution provides a robust path forward for complex, long-horizon robotic manipulation. Future work may involve scaling the high-level planner with massive pre-trained multimodal models to enhance common-sense reasoning in open-ended environments.

Summary updated

Abstract

Robotics agents often struggle to understand and follow the multi-modal prompts in complex manipulation scenes which are challenging to be sufficiently and accurately described by text alone. Moreover, for long-horizon manipulation tasks, the deviation from general instruction tends to accumulate if lack of intermediate guidance from high-level subgoals. For this, we consider can we generate subgoal images before act to enhance the instruction following in long-horizon manipulation with multi-modal prompts? Inspired by the great success of diffusion model in image generation tasks, we propose a novel hierarchical framework named as CoTDiffusion that incorporates diffusion model as a high-level planner to convert the general and multimodal prompts into coherent visual subgoal plans, which further guide the low-level policy model before action execution. We design a semantic alignment module that can anchor the progress of generated keyframes along a coherent generation chain, unlocking the chain-of-thought reasoning ability of diffusion model. Additionally, we propose bi-directional generation and frame concat mechanism to further enhance the fidelity of generated subgoal images and the accuracy of instruction following. The experiments cover various robotics manipulation scenarios including visual reasoning, visual rearrange, and visual constraints. CoTDiffusion achieves outstanding performance gain compared to the baselines without explicit subgoal generation, which proves that a subgoal image is worth a thousand words of instruction. The details and visualizations are available at https://cotdiffusion.github.io.
BibTeX
@inproceedings{ni2024generate,
  author = {Fei Ni and Jianye Hao and Shiguang Wu and Longxin Kou and Jiashun Liu and Yan Zheng and Bin Wang and Yuzheng Zhuang},
  title = {{Generate Subgoal Images Before Act: Unlocking the Chain-of-Thought Reasoning in Diffusion Model for Robot Manipulation with Multimodal Prompts}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2024},
  doi = {10.1109/cvpr52733.2024.01327},
}
Figure