Abstract
Masked Image Generation (MIG) offers a fast alternative to diffusion models by predicting image tokens in parallel. However, MIG often lags in image quality due to the semantic opacity of discrete visual tokens and the difficulty of learning long-range dependencies from scratch. We present KA-MIG (Knowledge-Augmented Masked Image Generation), a framework that enhances MIG by incorporating explicit knowledge graphs—co-occurrence, semantic similarity, and spatial incompatibility—extracted from training data. By injecting these priors through a graph-aware encoder and a lightweight fusion mechanism, KA-MIG achieves state-of-the-art results on ImageNet and MS-COCO benchmarks, demonstrating significant improvements in both FID and qualitative consistency.
Introduction
Masked Image Generation (MIG) has emerged as a compelling alternative to Diffusion models, primarily due to its ability to predict image tokens in parallel. This parallelization offers significant speed advantages during inference. However, a persistent gap remains: MIG models frequently struggle to match the high-fidelity image quality achieved by state-of-the-art diffusion-based systems.
The core challenge lies in the nature of the discrete visual tokens used in MIG. Unlike natural language processing, where words carry inherent semantic meaning, visual tokens—typically indices from a Vector Quantized Variational Autoencoder (VQ-VAE) codebook—are semantically opaque. Furthermore, representing an image as a grid (e.g., 256 tokens) introduces complex, long-range dependencies that are computationally difficult for a transformer to learn purely through self-attention from scratch.
To address these limitations, we introduce KA-MIG. This framework provides the generative model with a "semantic dictionary" by extracting and injecting explicit knowledge graphs from the training data, helping the model capture dependencies that are otherwise difficult to infer.
The KA-MIG Framework
The KA-MIG framework enhances the standard bidirectional transformer architecture by integrating external knowledge in three distinct phases: graph construction, graph-aware encoding, and a lightweight fusion mechanism.
1. Graph Construction
The foundation of KA-MIG is the extraction of three types of relational graphs from the training dataset:
- Co-occurrence Graph ($G_{co}$): This graph captures local spatial-semantic correlations by identifying which tokens frequently appear in proximity to one another.
- Semantic Similarity Graph ($G_s$): To overcome semantic opacity, the authors identify "synonymous" tokens. By comparing the spatial distributions of tokens across the entire dataset using Jensen-Shannon divergence, the framework identifies tokens that serve similar roles in image composition.
- Position-token Incompatibility Graph ($G_p$): This serves as a "negative prior." It identifies tokens that are statistically unlikely or impossible at specific positions for a given class (e.g., grass tokens should not appear in the center of an "airplane" body).
2. Graph-aware Encoder
Once constructed, these graphs are processed through Graph Convolutional Networks (GCNs). The GCNs transform the raw graph data into enriched token and position representations that can be effectively utilized by the generative model. This step ensures that the relational knowledge is translated into a high-dimensional space compatible with the transformer's internal embeddings.
3. Lightweight Fusion Mechanism
Integrating new knowledge into a pre-trained model without disrupting its existing capabilities is a non-trivial task. KA-MIG employs an additive-subtractive fusion strategy. Positive priors (co-occurrence and similarity) are added to the transformer layers to encourage likely token combinations. Conversely, negative priors (incompatibility) are subtracted to penalize implausible configurations.
To ensure stability, the framework utilizes "zero convolutions" that start with zero weights. This allows the model to integrate the external knowledge incrementally, preventing the new information from overwhelming the transformer's learned representations in the early stages of training.
Experimental Evaluation
The researchers evaluated KA-MIG across several high-resolution benchmarks, including ImageNet-256, ImageNet-512, and MS-COCO. The framework was tested on multiple backbone architectures, such as MaskGIT, AutoNAT, and TiTok, to demonstrate its model-agnostic nature.
Key Results
- State-of-the-Art Performance: On the ImageNet-256 benchmark, the TiTok-s128 backbone augmented with KA-MIG achieved a Fréchet Inception Distance (FID) of 1.90, a significant improvement over the baseline.
- Sequence Length Robustness: The most substantial gains were observed in models with longer token sequences (e.g., MaskGIT), confirming that explicit knowledge graphs are particularly effective at mitigating the difficulty of learning long-range dependencies.
- Qualitative Improvements: Images generated with KA-MIG exhibit sharper details and improved semantic consistency. The reduction in "hallucinations" is largely attributed to the effective use of the negative prior.
Discussion and Insights
A key insight from this work is the "Power of No." While most generative research focuses on what a model should generate, KA-MIG demonstrates that explicitly defining what should not be generated (via the incompatibility graph) is a powerful tool for improving structural coherence. Furthermore, because the knowledge graphs are static and extracted from the training set, they can be precomputed, allowing the performance boost to come with negligible computational overhead during image generation.
Limitations and Future Work
The Position-token Incompatibility Graph ($G_p$) currently relies on class labels, which may limit its application to unconditioned generation. Additionally, scaling these graphs to massive, web-scale datasets beyond ImageNet may pose memory challenges. Future research could explore dynamic graph structures that update during the generation process to provide even finer control over image synthesis.
Summary updated