PONDER: Learning to Generate in the Right Order

Julius Erbach, Dominik Narnhofer, Bernt Schiele, Jan Eric Lenssen, Konrad SchindlerarXiv 2026 · arXiv

TL;DR

PONDER (Policy for Ordering, Next-element DEcisions and Rethinking) is a reinforcement learning framework that teaches generative models the optimal sequence for filling in image patches. By introducing a "rethinking" mechanism that allows the model to revise earlier, inconsistent predictions as more context becomes available, PONDER significantly improves the quality and logical consistency of frozen generative models across tasks ranging from ImageNet synthesis to solving visual Sudokus.

What Problem Does This Paper Address?

Most modern masked generative models (like MAR or MaskGIT) create images by iteratively filling in missing patches. However, the order in which these patches are generated is usually determined by simple heuristics: a fixed raster scan, a random permutation, or a "greedy" approach that prioritizes patches the model is most confident about.

The authors argue that these strategies are suboptimal. Generation is a sequential decision-making process where early choices dictate the context for everything that follows. A "greedy" choice might be easy for the model to make now but could lead to a global inconsistency later—a "dead end" in the generation process. Furthermore, in standard models, once a patch is generated, it is set in stone. If the model realizes ten steps later that a previously generated patch doesn't fit the emerging scene, it has no way to fix it.

Methods

PONDER treats generation order as a learnable policy. The framework consists of a frozen generator and a trainable ordering policy (a bidirectional transformer).

Key innovations include:

  1. Learned Ordering: Instead of a fixed schedule, the policy observes the current state of the canvas and the generator's internal features to decide which patch to fill next.
  2. Rethinking: This is the "backspace" of image generation. The policy can choose to "re-mask" a previously filled patch, effectively discarding it. The generator then fills that patch again later, benefiting from the additional context provided by the rest of the generated image.
  3. FIDgrad Reward: Training a policy to improve image quality is hard because metrics like Fréchet Inception Distance (FID) are calculated over entire distributions, not single images. The authors introduced FIDgrad, a novel reward function that estimates the first-order effect a single generated sample has on the overall FID of the distribution. This allows them to use standard policy gradient methods (specifically Group-Relative Policy Gradients or GRPO) to optimize for distribution-level quality.
Overview of the PONDER framework showing the policy selecting patches to fill or re-mask based on terminal rewards.

Datasets

The authors evaluated PONDER across three distinct generative tasks:

  • Visual Sudoku (MNIST-based): A challenging task where the model must generate a valid Sudoku grid of handwritten digits. This tests global constraint satisfaction.
  • ImageNet-256: Class-conditional image generation using the MAR-B model.
  • Text-to-Image Synthesis: Using the MaskGen-KL-XL model, evaluated on the HPSv2 (Human Preference Score) and GenEval benchmarks.

Key Insights and Ideas

  • Order as Strategy: The paper demonstrates that generation order isn't just a technical detail—it's a strategic component of the generative process. By learning to "look ahead," the policy can establish a global structure early on, making subsequent steps easier and more accurate.
  • The Power of Revision: The "rethinking" mechanism is a game-changer for consistency. In the Sudoku task, a single wrong digit early on makes a valid solution impossible. Rethinking allows the model to detect these "rule breaks" and correct them.
  • Test-Time Scaling: A fascinating property of PONDER is that even with a fixed policy, increasing the "rethinking budget" (the number of allowed revisions) at inference time leads to monotonic improvements in quality.
Visual Sudoku example showing how the policy identifies a rule break, re-masks the cell, and regenerates the correct number.

Key Results

PONDER achieves impressive gains while keeping the underlying generative models completely frozen:

  • Visual Sudoku: On the "hard" split, accuracy skyrocketed from a baseline of 3.9% (random order) to 56.6% with learned ordering, and finally to 96.7% with rethinking.
  • ImageNet: PONDER reduced the FID of the MAR-B model from 4.09 to 2.70, a substantial leap in image quality.
  • Text-to-Image: On the HPSv2 benchmark, PONDER improved the preference score from 24.50 to 26.95, outperforming random ordering and greedy baselines.
Figure 1: Learning to generate and rethink. Left: a learned policy selects positions of patches to generate or mask for later regeneration, enabling revision of earlier predictions. Right: increasing the inference-time rethinking budget improves Sudoku-hard accuracy and MAR-B image quality, with each policy and generator held fixed.

Limitations and Future Work

While powerful, PONDER does introduce a small computational overhead, as it requires a forward pass of the policy network for each generation iteration. Additionally, the current rethinking budget is a fixed parameter; the authors suggest that learning a dynamic "stopping condition"—where the model decides for itself when it has "thought enough"—is a promising next step.

Finally, the success of the method is heavily tied to the quality of the reward function. While FIDgrad works well for general image quality, more complex tasks might require even more sophisticated, multi-faceted reward signals.

Summary updated

Abstract

We introduce PONDER, a method that learns in which order an image generator should fill in content and which of its earlier predictions to revise. Masked generative models construct a sample by repeatedly filling in subsets of missing variables. The order of these partial predictions determines the context available in subsequent steps and therefore affects the quality of the final sample. Common strategies are to choose the order randomly or based on local confidence estimates. Conditioned on the partially generated sample and the generator's features, our policy chooses which positions to fill next and/or which previously filled positions to mask for later regeneration, a mechanism we call rethinking. In this way, the generation process is able to adapt to the evolving context and also to revise earlier predictions. We optimize the policy using group-relative policy gradients, with rewards that measure constraint satisfaction, human preference, or distribution quality. For the latter, we introduce FIDgrad, an image-level reward derived from the (first-order) change in Fréchet Inception Distance caused by adding a generated sample to the current feature distribution. With the generators frozen, learned ordering reduces MAR-B ImageNet FID from 4.09 to 2.78 and raises the accuracy on hard visual Sudokus from 3.9\% to 56.6\%, surpassing uncertainty-based greedy ordering (38.7\%). Rethinking further improves these results to 2.70 FID and 96.7\% accuracy, while increasing the rethinking budget of a fixed policy yields continued improvements at inference time.
BibTeX
@misc{erbach2026ponder,
  author = {Julius Erbach and Dominik Narnhofer and Bernt Schiele and Jan Eric Lenssen and Konrad Schindler},
  title = {{PONDER: Learning to Generate in the Right Order}},
  year = {2026},
  eprint = {2610.23531},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2610.23531},
}
Figure