Shape of Thought: Progressive Object Assembly via Visual Chain-of-Thought

Yu Huo, Siyu Zhang, Kun Zeng, Haoyue Liu, Owen Lee, Junlin chen, Yuquan Lu, Yifu Guo, Yaodong Liang, Xiaoying TangICML 2026 · OpenReview

Abstract

Multimodal models for text-to-image generation have achieved strong visual fidelity, yet they remain brittle under compositional structural constraints—notably generative numeracy, attribute binding, and part-level relations. To address these challenges, we propose **Shape-of-Thought (SoT)**, a visual CoT framework for *process-supervised progressive shape assembly in the rendered 2D domain*, without external engines at inference time. SoT trains a unified multimodal autoregressive model to generate interleaved textual plans and rendered intermediate states, helping the model capture shape-assembly logic without producing explicit geometric representations. Unlike text-only CoT, each decision is grounded in a rendered state, making counts, attachments, topology, and intermediate part-addition errors inspectable across the trajectory. To support this paradigm, we introduce **SoT-26K**, a large-scale dataset of grounded assembly traces derived from part-based CAD hierarchies, and **T2S-CompBench**, a benchmark for evaluating structural integrity and trace faithfulness. Fine-tuning on SoT-26K achieves 88.4\% on component numeracy and 84.8\% on structural topology, outperforming direct generation by +24.2 points on component numeracy and +19.3 points on structural topology. SoT establishes a transparent testbed for rendered-domain structure-aware generation. The code is available at https://github.com/yuhuo03/Shape-of-Thought.
BibTeX
@inproceedings{huo2026shape,
  author = {Yu Huo and Siyu Zhang and Kun Zeng and Haoyue Liu and Owen Lee and Junlin chen and YUQUAN LU and Yifu Guo and Yaodong Liang and Xiaoying Tang},
  title = {{Shape of Thought: Progressive Object Assembly via Visual Chain-of-Thought}},
  booktitle = {International Conference on Machine Learning},
  year = {2026},
  url = {https://openreview.net/forum?id=ZsJnkEgh6I},
}

Loading related content...

Figure