Visual Chain of Thought: Bridging Logical Gaps with Multimodal Infillings

Daniel Philip Rose, Vaishnavi Himakunthala, Andy Ouyang, Ryan He, Alex Mei, Yujie Lu, Michael Saxon, Chinmay Sonar, Diba Mirza, William Yang WangSubmission to ICLR 2024 · Withdrawn

TL;DR

Visual Chain-of-Thought (VCoT) extends reasoning prompts into the multimodal realm by recursively generating synthetic image-text pairs to bridge the logical gaps in sequential data. Using off-the-shelf foundation models guided by global context and CLIP similarity, VCoT creates coherent intermediate steps that improve downstream tasks like visual storytelling and instruction summarization while offering human-interpretable reasoning paths.

What Problem Does This Paper Address?

Standard Chain-of-Thought (CoT) prompting decomposes complex reasoning into intermediate steps, but it remains predominantly text-only and focused on question-answering. In sequential vision-language tasks—such as visual storytelling, instruction following, or step-by-step navigation—input sequences (e.g., sparse keyframes or bulleted steps) often have large logical leaps between consecutive elements.

Text-only models struggle to bridge these jumps because language alone misses visual spatial context. Conversely, purely visual generators often imagine irrelevant details that drift away from the overarching narrative. The authors tackle this challenge: How can models bridge logical gaps in sequential data by generating multimodal intermediate steps?

Methods

VCoT is a training-free, pipeline-based framework that integrates off-the-shelf pretrained models: GPT-3.5 (text-davinci-003), Stable Diffusion (v1.4), OFA-base (image captioning), and CLIP.

Input Sequence (Text / Images) 
  ↳ 1. Task Unification (Convert all inputs to text-visual pairs)
  ↳ 2. Multipoint Foveation (Extract global context/focus via GPT-3.5)
  ↳ 3. Recursive Infilling (Generate intermediate steps to depth=2)
      ├─ Candidate Text Generation (GPT-3.5) → CLIP Selection
      └─ Candidate Image Generation (Stable Diffusion) → CLIP Selection
  ↳ Enhanced Sequence for Downstream Tasks
  1. Task Unification: If an input dataset is text-only, VCoT uses Stable Diffusion to generate candidate visuals and picks the top match via CLIP similarity, formatting all inputs into uniform text-image pairs $(v_i, t_i)$.
  2. Multipoint Foveation (MPF): Local pairwise infilling easily overfits to isolated details. To maintain global coherence, VCoT captions input images, feeds the entire sequence to GPT-3.5 with few-shot exemplars, and extracts a sequence-level "global focus" (e.g., "A romantic scene of a couple near a lighthouse").
  3. Novelty-Driven Recursive Infilling: Between pairs $(v_{i-1}, t_{i-1})$ and $(v_{i+1}, t_{i+1})$, VCoT recursively generates intermediate infillings up to a fixed recursion depth (depth-limit = 2).
  4. Consistency-Driven Visual Augmentation:
  5. Text Infilling: GPT-3.5 proposes five candidates conditioned on surrounding captions, text, and the global focus. CLIP embeddings select the candidate most aligned with surrounding context.
  6. Visual Infilling: Stable Diffusion generates four candidate images from the selected text, and CLIP selects the most consistent image.

Datasets

The authors evaluate VCoT on two sequential benchmarks:

  • VIST (Visual Storytelling): 2,021 test stories, each containing a sequence of five image-text pairs derived from Flickr albums.
  • WikiHow: 6,000 test set "How-To" articles. Used as a text-only procedural dataset to test VCoT's task unification and instruction summarization.

Key Insights and Ideas

  • Multimodal grounding beats parallel unimodal chains: Simply running a text chain (CoT) and an image chain (Chain-of-Images / CoI) independently leads to visual hallucinations that diverge from the narrative. Conditioning image synthesis directly on logically aligned text candidates keeps both modalities anchored.
  • Global foveation prevents narrative derailment: Without sequence-level focus (MPF), generative models suffer from local myopia (e.g., hyper-focusing on an incidental tattoo on an actor's leg instead of advancing the main story about touring a city).
  • Intermediate steps act as synthetic data augmentation: Beyond interpretability, inserting synthetic multimodal infillings provides richer context that directly improves downstream generation quality.

Key Results

Evaluation relied on extensive crowdsourcing via Amazon Mechanical Turk (~20,500 workers across 7,062 infillings and hundreds of downstream examples) measuring consistency, novelty, coherence (VIST), and descriptiveness (WikiHow).

  • Infilling Quality: In head-to-head win-tie-loss evaluations against parallel CoT + CoI, VCoT consistently won:
  • Image Consistency: 26.82% Win vs. 20.16% Loss (53.02% Tie)
  • Text Consistency: 28.07% Win vs. 19.73% Loss (52.21% Tie)
  • Image Novelty: 30.13% Win vs. 19.63% Loss (50.24% Tie)
  • Text Novelty: 25.77% Win vs. 21.37% Loss (52.86% Tie)
  • Downstream Task Performance:
  • On WikiHow, VCoT outperformed the CoT baseline in average win rate (29.20% win vs. 26.48% loss), producing more descriptive and novel instructions.
  • On VIST, VCoT achieved higher average win rates against CoT (35.05% win vs. 30.77% loss), particularly boosting story coherence (35.11% win vs. 29.79% loss).

Limitations and Future Work

  • Fixed Recursion Depth: Dynamic halting by prompting GPT-3.5 to detect remaining logical gaps proved noisy and inconsistent. The authors defaulted to a fixed depth of 2; learning optimal, adaptive stopping criteria remains open.
  • Evaluation Scalability: Assessment heavily relies on human annotators due to a lack of reliable automated metrics for multimodal sequence coherence.
  • Repetition vs. Novelty Trade-offs: Heavy CLIP filtering occasionally causes lexical and visual repetition across adjacent infillings.
  • Future Applications: Extending VCoT to domains requiring fine-grained temporal understanding, such as procedural planning, video comprehension, and biological sequence modeling.

Summary updated

Abstract

Recent advances in large language models elicit reasoning in a chain-of-thought that allows models to decompose problems in a human-like fashion. Though this paradigm improves multi-step reasoning ability in language models, it is limited by being unimodal and applied mainly to question-answering tasks. We claim that incorporating visual augmentation into reasoning is essential, especially for complex, imaginative tasks. Consequently, we introduce VCoT, a novel method that leverages chain-of-thought prompting with vision-language grounding to recursively bridge the logical gaps within sequential data. Our method uses visual guidance to generate synthetic multimodal infillings that add consistent and novel information to reduce the logical gaps for downstream tasks that can benefit from temporal reasoning, as well as provide interpretability into models' multi-step reasoning. We apply VCoT to the Visual Storytelling and WikiHow summarization datasets and demonstrate through human evaluation that VCoT offers novel and consistent synthetic data augmentation beating chain-of-thought baselines, which can be used to enhance downstream performance.
BibTeX
@misc{rose2023visual,
  author = {Daniel Philip Rose and Vaishnavi Himakunthala and Andy Ouyang and Ryan He and Alex Mei and Yujie Lu and Michael Saxon and Chinmay Sonar and Diba Mirza and William Yang Wang},
  title = {{Visual Chain of Thought: Bridging Logical Gaps with Multimodal Infillings}},
  year = {2023},
  eprint = {2305.02317},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2305.02317},
}
Figure