Introduction
Robotic agents frequently struggle with long-horizon tasks that involve complex, multimodal instructions. For instance, a command like "Put the [image of a block] into the [image of a container] without crossing the [image of a line]" requires both spatial reasoning and temporal planning. Traditional end-to-end models often suffer from error accumulation, while LLM-based planners that decompose tasks into text-based steps often fail to capture the nuanced spatial constraints necessary for physical grounding.
The core insight of this work is that "a picture is worth a thousand words." By generating a sequence of mental subgoal images—a visual "Chain-of-Thought" (CoT)—robots can decompose complex prompts into achievable visual milestones, significantly improving success rates in long-horizon manipulation tasks.
The CoTDiffusion Framework
CoTDiffusion is a hierarchical framework designed to bridge the gap between high-level planning and low-level execution. It consists of two primary components: a high-level visual planner and a low-level action policy.
Visual Chain-of-Thought (CoT)
Instead of predicting motor actions directly from a complex prompt, a conditional diffusion model generates a sequence of keyframes. These keyframes serve as intermediate visual goals, effectively breaking a long-horizon task into manageable segments. This approach allows the model to "think" in the visual domain before acting.
Semantic Alignment Module (SAM)
To ensure the diffusion model generates the correct next step, the Semantic Alignment Module (SAM) acts as a progress tracker. It utilizes a "triple alignment" architecture that compares the initial observation, the current state, and the multimodal prompt to determine the current stage of the task. This module ensures that the generated subgoals are tightly grounded in the original instructions.
Training and Consistency
The model is trained in two stages. First, the SAM undergoes "coarse alignment" via a masked patch prediction task to learn which parts of a scene change between subgoals. Second, the diffusion model is trained using bi-directional generation, requiring it to predict both the next subgoal and reconstruct the previous one. This ensures logical consistency and prevents the "hallucinations" common in standard generative models.
Experimental Evaluation
The researchers evaluated CoTDiffusion on the VIMA-BENCH, focusing on three challenging categories: Visual Rearrange (matching a specific setup), Visual Reasoning (logic-based instructions), and Visual Constraints (adhering to spatial boundaries).
Key Results
CoTDiffusion demonstrated superior performance compared to state-of-the-art baselines, including Gato, Flamingo, VIMA, and SuSIE.
As shown in the results, CoTDiffusion achieved an overall success rate of 64.6%, significantly outperforming VIMA (49.5%). Notably, it maintained strong performance even in "L3" tasks involving novel object-task combinations, where other models struggled. Comparisons with SuSIE further highlighted that generated images are more effective for robot planning than human-written text sub-prompts, as SuSIE's performance lagged even when provided with manual text decompositions.
Conclusion
By treating image generation as a visual Chain-of-Thought, CoTDiffusion allows robots to plan effectively in the visual domain. This decoupling of high-level planning from low-level execution provides a robust path forward for complex, long-horizon robotic manipulation. Future work may involve scaling the high-level planner with massive pre-trained multimodal models to enhance common-sense reasoning in open-ended environments.
Summary updated