Reject Decoding via Language-Vision Models for Text-to-Image Synthesis

Fuxiang Wu, Liu Liu, Fusheng Hao, Fengxiang He, Lei Wang, Jun Cheng

Abstract

Autoregressive text-to-image transformers often suffer from error propagation and computational inefficiency during inference. We present Reject Decoding, a novel strategy that uses lightweight vision-language models to prune low-quality generation paths early. This approach allows for a broader search of the candidate space within the same computational budget, significantly improving image quality and semantic alignment.

Introduction

Modern text-to-image synthesis relies heavily on autoregressive transformers that predict image tokens sequentially. While effective, this method is prone to compounding errors: a single "bad" token early in the sequence can derail the entire generation. To mitigate this, standard practice involves generating a large batch of images and re-ranking them using a model like CLIP. However, this "generate-then-rank" paradigm is inefficient, as it fully decodes many images that are eventually discarded.

Methodology: Reject Decoding

Reject Decoding addresses these inefficiencies by introducing early-exit points during the decoding process. The algorithm is built on three pillars:

  1. Tiny Multimodal Scorers: The authors developed a suite of "tiny" vision-language transformers (e.g., 8 layers). These models are specifically trained to evaluate the alignment between a text prompt and a partial set of image tokens, providing a signal of generation quality long before the image is complete.
  2. Early Pruning: Instead of decoding a fixed number of paths to completion, the process begins with a larger initial pool. At specific intervals, the tiny scorers evaluate the partial paths, and those falling below a calculated threshold are rejected.
  3. Search Space Expansion: By terminating poor paths early, the algorithm reallocates computational resources to explore more initial candidates. This effectively increases the model's "imagination" at the critical early stages of synthesis.

To maintain computational parity with standard decoding, the authors introduced an automated thresholding algorithm. This ensures that the number of paths kept at each stage balances the cost of the tiny scorers against the savings from pruned decoding steps, optimizing the "compute-to-quality" ratio.

Visual representation of the Reject Decoding process, showing how paths are evaluated and pruned at different decoding steps to reach high-quality final images.

Experimental Results

The method was evaluated using the RQ-Transformer architecture on MS-COCO and large-scale datasets. Using a 3.9B parameter model, Reject Decoding demonstrated superior performance over standard re-ranking:

  • Visual Quality: The Inception Score (IS) increased by 1.27, and the FID decreased by 2.11, indicating more realistic and diverse images.
  • Semantic Consistency: R-precision improved by 10.97%, showing that the final images are much more closely aligned with the input text prompts.
  • Efficiency: Total inference time for the 3.9B model was reduced by 4.32 seconds, as the system avoided the costly VQ-GAN decoding steps for rejected paths.

The authors also found that the "tiny" scorers, despite their small size, were remarkably resilient. Even with only a fraction of tokens generated, the scorers could reliably distinguish between paths that would lead to high-quality alignment and those that would fail, validating the core premise of early-stage intervention.

Discussion and Future Work

A key insight of this work is that partial image information is often sufficient to identify when a generation is failing. Lightweight models can provide a robust "rejection" signal, allowing the main transformer to focus on high-potential paths.

Future work may involve integrating vision-only models to monitor structural integrity or exploring more complex multimodal architectures to guide the decoding process. The dependency on scorer quality remains a limitation, as biased scorers could potentially prune creative or unconventional but valid paths.

Summary updated

Abstract

Transformer-based text-to-image synthesis generates images from abstractive textual conditions and achieves prompt results. Since transformer-based models predict visual tokens step by step in testing, where the early error is hard to be corrected and would be propagated. To alleviate this issue, the common practice is drawing multi-paths from the transformer-based models and re-ranking the multi-images decoded from multi-paths to find the best one and filter out others. Therefore, the computing procedure of excluding images may be inefficient. To improve the effectiveness and efficiency of decoding, we exploit a reject decoding algorithm with tiny multi-modal models to enlarge the searching space and exclude the useless paths as early as possible. Specifically, we build tiny multi-modal models to evaluate the similarities between the partial paths and the caption at multi scales. Then, we propose a reject decoding algorithm to exclude some lowest quality partial paths at the inner steps. Thus, under the same computing load as the original decoding, we could search across more multi-paths to improve the decoding efficiency and synthesizing quality. The experiments conducted on the MS-COCO dataset and large-scale datasets show that the proposed reject decoding algorithm can exclude the useless paths and enlarge the searching paths to improve the synthesizing quality by consuming less time.
BibTeX
@inproceedings{wu2023reject,
  author = {Fuxiang Wu and Liu Liu and Fusheng Hao and Fengxiang He and Lei Wang and Jun Cheng},
  title = {{Reject Decoding via Language-Vision Models for Text-to-Image Synthesis}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2023},
  doi = {10.1609/aaai.v37i3.25379},
}
Figure