Image Content Generation with Causal Reasoning

Xiaochuan Li, Baoyu Fan, Runze Zhang, Liang Jin, Di Wang, Zhenhua Guo, Yaqian Zhao, Rengang LiAAAI 2024 · DOI

Abstract

While state-of-the-art generative AI models excel at literal text-to-image translation, they often struggle with the underlying logic of cause and effect. This paper introduces a new challenge: Visual Question Answering with Image (VQAI). To address this, the authors propose Latent Guided Diffusion (LGD), a framework that leverages the reasoning capabilities of Large Language Models (LLMs) to guide image synthesis. By training on a novel dataset derived from Tom and Jerry, the model learns to infer implicit consequences—such as physical breakage or emotional reactions—that standard diffusion models typically overlook.

The Challenge of Causal Reasoning

Current image generators like Stable Diffusion or DALL-E are highly proficient at drawing what is explicitly stated in a prompt. However, they lack a "world model" for causal reasoning. For instance, if presented with an image of a glass falling and asked to generate the next state, a standard model might simply render the glass floating near the floor. It fails to infer the logical conclusion: the glass shattering and the liquid spilling.

A successful causal generation showing a broken glass and spilled juice.

The authors argue that while LLMs have demonstrated reasoning in the text domain, visual models require a new architecture to handle what they term a visual syllogism:

  1. Major Premise: The initial image (the current state).
  2. Minor Premise: A question containing a condition (the "if" factor).
  3. Conclusion: A generated image representing the logical outcome.

The VQAI Task and Dataset

To benchmark this capability, the researchers introduced the VQAI Dataset, consisting of 17,524 image pairs from 755 episodes of Tom and Jerry. This cartoon was chosen because its world follows exaggerated but consistent physical rules, and its storytelling is primarily visual rather than dialogue-driven. The task requires the model to look at an initial frame, process a conditional question, and generate the resulting state.

An example of the VQAI task: predicting the visual outcome of pulling a lamp cord.

Proposed Method: Latent Guided Diffusion (LGD)

The core innovation of this work is the Latent Guided Diffusion (LGD) model. Unlike previous "Answer-Guided" approaches that first generate a text description and then an image (a process that loses fine-grained visual detail), LGD performs its reasoning within the latent space.

Technical Architecture

  • Frozen LLM Backbone: The model utilizes Flan-T5-XXL to perform logical reasoning.
  • Latent Space Translation (LST): Because the LLM and the Diffusion model operate in different feature spaces, the LST module acts as a bridge, translating multimodal features extracted by a Q-Former into a format the diffusion process can use.
  • Contrast Causal Predictive Coding (CCPC): This module employs contrastive learning to help the model distinguish between a logically correct causal outcome and an unrelated scene.
  • Multimodal Causal Chain Supervision (MCCS): The model is trained on "causal chains"—step-by-step logical annotations (e.g., Jerry kicks foot → Dog feels pain → Dog shows painful expression). This allows the model to account for "invisible" states like emotions or physical sensations.

Key Results and Insights

The LGD model significantly outperformed standard baselines in both automated metrics and human evaluation. Human reviewers found LGD's generations to be semantically correct 32.39% of the time, nearly doubling the performance of the Answer-Guided baseline (18.52%). Furthermore, in user preference tests, LGD was chosen as the superior model in 51.35% of cases.

A critical insight from the study is that text is a "lossy" bridge. By keeping the reasoning process in the latent space, LGD preserves the "infinite granularity" of the original scene, ensuring that the generated outcome remains visually consistent with the starting image while adhering to causal logic.

Limitations and Future Work

While LGD shows promise, it is currently limited by the cartoon domain of the VQAI dataset. Transitioning these causal reasoning abilities to photorealistic, real-world scenarios remains a significant hurdle. Additionally, the model still struggles with "long-tail" events (rare causal occurrences) and highly complex, multi-step reasoning chains. Future research will likely focus on deeper integration of chain-of-thought reasoning directly into the visual decoding process to handle more sophisticated scenarios.

Summary updated

BibTeX
@inproceedings{li2024image,
  author = {Xiaochuan Li and Baoyu Fan and Runze Zhang and Liang Jin and Di Wang and Zhenhua Guo and Yaqian Zhao and Rengang Li},
  title = {{Image Content Generation with Causal Reasoning}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2024},
  doi = {10.1609/aaai.v38i12.29269},
}
Figure