Improved Masked Image Generation with Knowledge-Augmented Token Representations

Guotao Liang, Baoquan Zhang, Zhiyuan Wen, Zihao Han, Yunming YearXiv 2025 · DOI · arXiv

Abstract

Masked Image Generation (MIG) offers a fast alternative to diffusion models by predicting image tokens in parallel. However, MIG often lags in image quality due to the semantic opacity of discrete visual tokens and the difficulty of learning long-range dependencies from scratch. We present KA-MIG (Knowledge-Augmented Masked Image Generation), a framework that enhances MIG by incorporating explicit knowledge graphs—co-occurrence, semantic similarity, and spatial incompatibility—extracted from training data. By injecting these priors through a graph-aware encoder and a lightweight fusion mechanism, KA-MIG achieves state-of-the-art results on ImageNet and MS-COCO benchmarks, demonstrating significant improvements in both FID and qualitative consistency.

Introduction

Masked Image Generation (MIG) has emerged as a compelling alternative to Diffusion models, primarily due to its ability to predict image tokens in parallel. This parallelization offers significant speed advantages during inference. However, a persistent gap remains: MIG models frequently struggle to match the high-fidelity image quality achieved by state-of-the-art diffusion-based systems.

The core challenge lies in the nature of the discrete visual tokens used in MIG. Unlike natural language processing, where words carry inherent semantic meaning, visual tokens—typically indices from a Vector Quantized Variational Autoencoder (VQ-VAE) codebook—are semantically opaque. Furthermore, representing an image as a grid (e.g., 256 tokens) introduces complex, long-range dependencies that are computationally difficult for a transformer to learn purely through self-attention from scratch.

Comparison of standard Masked Image Generation (MIG) training and inference versus the proposed Knowledge-Augmented (KA-MIG) approach.

To address these limitations, we introduce KA-MIG. This framework provides the generative model with a "semantic dictionary" by extracting and injecting explicit knowledge graphs from the training data, helping the model capture dependencies that are otherwise difficult to infer.

The KA-MIG Framework

The KA-MIG framework enhances the standard bidirectional transformer architecture by integrating external knowledge in three distinct phases: graph construction, graph-aware encoding, and a lightweight fusion mechanism.

The three-step KA-MIG pipeline: Graph Construction, Graph-aware Encoding, and Lightweight Fusion Mechanism.

1. Graph Construction

The foundation of KA-MIG is the extraction of three types of relational graphs from the training dataset:

  • Co-occurrence Graph ($G_{co}$): This graph captures local spatial-semantic correlations by identifying which tokens frequently appear in proximity to one another.
  • Semantic Similarity Graph ($G_s$): To overcome semantic opacity, the authors identify "synonymous" tokens. By comparing the spatial distributions of tokens across the entire dataset using Jensen-Shannon divergence, the framework identifies tokens that serve similar roles in image composition.
  • Position-token Incompatibility Graph ($G_p$): This serves as a "negative prior." It identifies tokens that are statistically unlikely or impossible at specific positions for a given class (e.g., grass tokens should not appear in the center of an "airplane" body).

2. Graph-aware Encoder

Once constructed, these graphs are processed through Graph Convolutional Networks (GCNs). The GCNs transform the raw graph data into enriched token and position representations that can be effectively utilized by the generative model. This step ensures that the relational knowledge is translated into a high-dimensional space compatible with the transformer's internal embeddings.

3. Lightweight Fusion Mechanism

Integrating new knowledge into a pre-trained model without disrupting its existing capabilities is a non-trivial task. KA-MIG employs an additive-subtractive fusion strategy. Positive priors (co-occurrence and similarity) are added to the transformer layers to encourage likely token combinations. Conversely, negative priors (incompatibility) are subtracted to penalize implausible configurations.

To ensure stability, the framework utilizes "zero convolutions" that start with zero weights. This allows the model to integrate the external knowledge incrementally, preventing the new information from overwhelming the transformer's learned representations in the early stages of training.

Experimental Evaluation

The researchers evaluated KA-MIG across several high-resolution benchmarks, including ImageNet-256, ImageNet-512, and MS-COCO. The framework was tested on multiple backbone architectures, such as MaskGIT, AutoNAT, and TiTok, to demonstrate its model-agnostic nature.

Key Results

  • State-of-the-Art Performance: On the ImageNet-256 benchmark, the TiTok-s128 backbone augmented with KA-MIG achieved a Fréchet Inception Distance (FID) of 1.90, a significant improvement over the baseline.
  • Sequence Length Robustness: The most substantial gains were observed in models with longer token sequences (e.g., MaskGIT), confirming that explicit knowledge graphs are particularly effective at mitigating the difficulty of learning long-range dependencies.
  • Qualitative Improvements: Images generated with KA-MIG exhibit sharper details and improved semantic consistency. The reduction in "hallucinations" is largely attributed to the effective use of the negative prior.

Discussion and Insights

A key insight from this work is the "Power of No." While most generative research focuses on what a model should generate, KA-MIG demonstrates that explicitly defining what should not be generated (via the incompatibility graph) is a powerful tool for improving structural coherence. Furthermore, because the knowledge graphs are static and extracted from the training set, they can be precomputed, allowing the performance boost to come with negligible computational overhead during image generation.

Limitations and Future Work

The Position-token Incompatibility Graph ($G_p$) currently relies on class labels, which may limit its application to unconditioned generation. Additionally, scaling these graphs to massive, web-scale datasets beyond ImageNet may pose memory challenges. Future research could explore dynamic graph structures that update during the generation process to provide even finer control over image synthesis.

Summary updated

Abstract

Masked image generation (MIG) has demonstrated remarkable efficiency and high-fidelity images by enabling parallel token prediction. Existing methods typically rely solely on the model itself to learn semantic dependencies among visual token sequences. However, directly learning such semantic dependencies from data is challenging because the individual tokens lack clear semantic meanings, and these sequences are usually long. To address this limitation, we propose a novel Knowledge-Augmented Masked Image Generation framework, named KA-MIG, which introduces explicit knowledge of token-level semantic dependencies (\emph{i.e.}, extracted from the training data) as priors to learn richer representations for improving performance. In particular, we explore and identify three types of advantageous token knowledge graphs, including two positive and one negative graphs (\emph{i.e.}, the co-occurrence graph, the semantic similarity graph, and the position-token incompatibility graph). Based on three prior knowledge graphs, we design a graph-aware encoder to learn token and position-aware representations. After that, a lightweight fusion mechanism is introduced to integrate these enriched representations into the existing MIG methods. Resorting to such prior knowledge, our method effectively enhances the model's ability to capture semantic dependencies, leading to improved generation quality. Experimental results demonstrate that our method improves upon existing MIG for class-conditional image generation on ImageNet.
BibTeX
@misc{liang2025improved,
  author = {Guotao Liang and Baoquan Zhang and Zhiyuan Wen and Zihao Han and Yunming Ye},
  title = {{Improved Masked Image Generation with Knowledge-Augmented Token Representations}},
  year = {2025},
  eprint = {2511.12032},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2511.12032},
}
Figure