Let's Verify and Reinforce Image Generation Step by Step

Renrui Zhang, Chengzhuo Tong, Zhizheng Zhao, Ziyu Guo, Haoquan Zhang, Manyuan Zhang, Jiaming Liu, Peng Gao, Hongsheng LiCVPR 2025 · OpenReview

Abstract

Chain-of-Thought (CoT) reasoning has been extensively explored in large models to tackle complex understanding tasks. However, it still remains an open question whether such strategies can be applied to verifying and reinforcing image generation scenarios. In this paper, we provide the first comprehensive investigation into the potential of CoT reasoning to enhance autoregressive image generation. We focus on three techniques: scaling test-time computation for verification, aligning model preferences with Direct Preference Optimization (DPO), and integrating these techniques for complementary effects. Our results demonstrate that these approaches can be effectively adapted and combined to significantly improve image generation performance. Furthermore, given the pivotal role of reward models in our findings, we propose the Potential Assessment Reward Model (PARM) specialized for autoregressive image generation. PARM adaptively assesses each generation step through a potential assessment mechanism, merging the strengths of existing reward models. Using our investigated reasoning strategies, we enhance a baseline model, Show-o, to achieve superior results, with a significant +24% improvement on the GenEval benchmark, surpassing Stable Diffusion 3 by +15%.

Introduction

In the domain of Large Language Models (LLMs), models like OpenAI’s o1 have demonstrated that "thinking" longer at test-time—utilizing Chain-of-Thought (CoT) reasoning and reward models to verify intermediate steps—leads to superior performance in complex tasks like mathematics and coding.

Conversely, image generation has traditionally been treated as a "one-shot" process. Even modern autoregressive models, which generate images token-by-token, have struggled to leverage multi-step reasoning. The primary obstacle is the nature of partial visual data: unlike a mathematical proof where a partial step has logical validity, a partial image is often a blurry collection of pixels. Standard Process Reward Models (PRMs) used in text often fail here because they cannot distinguish between a "promising" blurry start and a "dead-end" one. This research bridges that gap by introducing verification and reinforcement strategies tailored for the visual domain.

Methodology

The researchers utilized Show-o, a state-of-the-art discrete autoregressive model, as their baseline. They explored three primary pillars to enhance generation:

1. Test-time Verification (Best-of-N)

Instead of selecting the first generated image, the system generates $N$ candidates and employs a reward model to select the optimal output. The study compared Outcome Reward Models (ORM), which evaluate the final product, against Process Reward Models (PRM), which evaluate intermediate generation steps.

2. Potential Assessment Reward Model (PARM)

PARM is the core innovation of this work. Standard PRMs struggle with the "blurry bottleneck" of early-stage autoregressive generation. PARM introduces an adaptive judging mechanism with three distinct tasks:

  • Clarity Judgment: It determines if an image is clear enough to be evaluated. If it is too blurry, the model waits.
  • Potential Assessment: Once clarity is reached, it assesses whether the current generation path has the potential to result in a high-quality final image.
  • Final Selection: It functions as an ORM to select the winner from the remaining high-potential paths.
Architectural comparison between Outcome Reward Models (ORM), Process Reward Models (PRM), and the proposed Potential Assessment Reward Model (PARM).

3. Reinforced Preference Alignment (DPO)

The researchers applied Direct Preference Optimization (DPO) and Iterative DPO to fine-tune the generator. This process "bakes" the reward model's preferences directly into the base model, allowing it to produce higher-quality images natively, reducing the reliance on heavy test-time searching.

Experimental Evaluation

The models were evaluated primarily on the GenEval benchmark, which tests complex prompt adherence involving counting, spatial relationships, and attribute binding. To train the reward models, the authors curated massive datasets, including 288K text-to-image ranking examples for ORM fine-tuning and 400K instances for the PARM architecture.

Key Results

  • Performance Gains: The baseline Show-o model's score of 53% was elevated to 77% through the combination of PARM and Iterative DPO—a +24% absolute improvement.
  • Competitive Benchmarking: This approach significantly outperformed industry leaders, beating Stable Diffusion 3 (63%) by 14% and surpassing DALL-E 3.
  • Scaling Laws: A critical finding was that performance scales consistently with test-time computation. As the number of samples ($N$) increases, the quality continues to climb, particularly when using PARM, whereas standard models tend to plateau earlier.
Performance comparison on the GenEval benchmark across different reward models and scaling factors (N).

Discussion and Insights

One of the most surprising findings was that ORM outperformed standard PRM for images. In LLM reasoning, PRMs are typically superior; however, for images, early steps are too blurry to judge accurately, and late steps are too visually similar to distinguish. PARM solves this by being "patient"—waiting for sufficient visual signal before passing judgment.

Furthermore, the study proved that DPO and Test-time Verification are complementary. DPO raises the baseline quality of the model, while verification (especially via PARM) provides the final polish. Combining these techniques yields the most robust results.

Limitations and Future Work

While effective, "Best-of-N" sampling is computationally expensive, requiring $N$ times the resources of a single generation. Future research may focus on more efficient search algorithms, such as tree search. Additionally, while this study focused on autoregressive models, adapting these step-by-step verification strategies to the denoising steps of Diffusion models remains a promising open challenge.

Summary updated

Abstract

Chain-of-Thought (CoT) reasoning has been extensively explored in large models to tackle complex understanding tasks. However, it still remains an open question whether such strategies can be applied to verifying and reinforcing image generation scenarios. In this paper, we provide the first comprehensive investigation in the potential of CoT reasoning to enhance autoregressive image generation. We focus on three techniques: scaling test-time computation for verification, aligning model preferences with Direct Preference Optimization (DPO), and integrating these techniques for complementary effects. Our results demonstrate that these approaches can be effectively adapted and combined to significantly improve image generation performance. Furthermore, given the pivotal role of reward models in our findings, we propose the Potential Assessment Reward Model (PARM) specialized for autoregressive image generation. PARM adaptively assesses each generation step through a potential assessment mechanism, merging the strengths of existing reward models. Using our investigated reasoning strategies, we enhance a baseline model, Show-o, to achieve superior results, with a significant +24% improvement on the GenEval benchmark, surpassing Stable Diffusion 3 by +15%. We hope our study provides unique insights and paves a new path for integrating CoT reasoning with autoregressive image generation. Code is released at https://github.com/ZiyuGuo99/Image-Generation-CoT.
BibTeX
@inproceedings{zhang2025let,
  author = {Renrui Zhang and Chengzhuo Tong and Zhizheng Zhao and Ziyu Guo and Haoquan Zhang and Manyuan Zhang and Jiaming Liu and Peng Gao and Hongsheng Li},
  title = {{Let's Verify and Reinforce Image Generation Step by Step}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2025},
  url = {https://openreview.net/forum?id=Z0yyPGOM4U},
}
Figure