Abstract
Generative image synthesis with diffusion models has recently achieved excellent visual quality in several tasks such as text-based or class-conditional image synthesis. Much of this success is due to a dramatic increase in the computational capacity invested in training these models. This work presents an alternative approach: inspired by its successful application in natural language processing, we propose to complement the diffusion model with a retrieval-based approach and to introduce an explicit memory in the form of an external database. During training, our diffusion model is trained with similar visual features retrieved via CLIP and from the neighborhood of each training instance. By leveraging CLIP's joint image-text embedding space, our model achieves highly competitive performance on tasks for which it has not been explicitly trained, such as class-conditional or text-image synthesis, and can be conditioned on both text and image embeddings. Moreover, we can apply our approach to unconditional generation, where it achieves state-of-the-art performance. Our approach incurs low computational and memory overheads and is easy to implement.
Introduction
The prevailing paradigm in high-fidelity image synthesis has been the development of increasingly large parametric models. These models attempt to compress the vast complexity of visual datasets into billions of trainable weights. While successful, this "scaling law" approach introduces significant resource intensity and a lack of flexibility; once trained, a model's knowledge is frozen within its weights, requiring costly retraining or fine-tuning to adapt to new domains.
This paper proposes a shift toward semi-parametric neural image synthesis. Rather than forcing a model to memorize every visual concept, we provide it with an external database to "look up" relevant information. By combining a relatively small generative backbone with a retrieval mechanism, we demonstrate that models can achieve state-of-the-art performance with fewer parameters and gain the unique ability to generalize to new styles—such as artistic rendering—simply by swapping the underlying database.
Methodology: The RDM and RARM Framework
The core of this approach is the integration of a generative backbone (either a Diffusion Model or an Autoregressive Model) with an external image database. The authors introduce two primary architectures: Retrieval-Augmented Diffusion Models (RDM) and Retrieval-Augmented Autoregressive Models (RARM).
Retrieval via CLIP
To bridge the gap between different modalities, the system utilizes CLIP (Contrastive Language-Image Pre-training). During training, for any given image, the system retrieves the $k$ most similar images from a 20-million-entry database (derived from OpenImages patches) based on CLIP embeddings. At inference, this same mechanism allows for text-to-image synthesis by using a text prompt to query the image database.
Conditioning and Sampling
The retrieved "nearest neighbors" are integrated into the generative process via a cross-attention mechanism. This allows the model to learn how to compose a scene by attending to the content and style of the retrieved examples. To further refine the output, the authors introduce Top-m Sampling, a strategy that allows users to navigate the trade-off between sample quality (precision) and diversity (recall) by truncating the distribution of retrieved neighbors.
Experimental Results
The study evaluated the RDM and RARM frameworks across several benchmarks, including ImageNet, FFHQ, and MS-COCO.
- Efficiency and Quality: On ImageNet 256x256, the RDM achieved a Fréchet Inception Distance (FID) of 11.31. This significantly outperforms the unconditional state-of-the-art ADM model (FID 26.21) while utilizing a smaller parameter count (400M vs. 554M).
- Label-Free Conditioning: A significant advantage of this framework is its ability to perform class-conditional and text-to-image synthesis without being explicitly trained on those tasks. By using text prompts to query the CLIP-based retrieval system, the model can generate specific content based on the visual features of the retrieved neighbors.
- Scalability: The performance gap between semi-parametric and fully-parametric models was found to increase as the complexity of the dataset grew, suggesting that retrieval-augmentation is particularly beneficial for diverse, large-scale data.
Post-hoc Domain Transfer
One of the most compelling features of the retrieval-augmented approach is zero-shot stylization. Because the model is conditioned on the external database rather than fixed weights, its behavior can be modified after training. By replacing the standard ImageNet database with a specialized database like WikiArt, the model begins generating artistic interpretations of concepts it has learned, effectively transferring its knowledge to a new domain without any additional training.
Discussion and Limitations
While the retrieval-augmented approach offers significant gains in flexibility and efficiency, certain challenges remain:
- Inference Speed: Although the retrieval step itself is highly efficient (under 1ms), the iterative nature of diffusion and autoregressive models remains a bottleneck compared to single-pass models like GANs.
- Storage Requirements: Maintaining a large-scale retrieval database requires significant memory (approximately 2GB per million images for CLIP embeddings).
- Embedding Dependency: The model's performance is inherently tied to the quality of the CLIP embedding space. Ambiguities or biases within CLIP will inevitably be reflected in the generative output.
Conclusion
Retrieval-Augmented Diffusion Models represent a promising alternative to the trend of massive parametric scaling. By separating content retrieval from generative composition, these models achieve superior image quality and unprecedented flexibility. Future work will explore the application of these methods to even larger databases and investigate whether the scaling laws observed in large language models apply similarly to semi-parametric visual synthesis.
Summary updated