Context Canvas: Enhancing Text-to-Image Diffusion Models with Knowledge Graph-Based RAG

Kavana Venkatesh, Yusuf Dalva, Ismini Lourentzou, Pinar Yanardag

Abstract

We introduce a novel approach to enhance the capabilities of text-to-image models by incorporating a graph-based RAG. Our system dynamically retrieves detailed character information and relational data from the knowledge graph, enabling the generation of visually accurate and contextually rich images. This capability significantly improves upon the limitations of existing T2I models, which often struggle with the accurate depiction of complex or culturally specific subjects due to dataset constraints. Furthermore, we propose a novel self-correcting mechanism for text-to-image models to ensure consistency and fidelity in visual outputs, leveraging the rich context from the graph to guide corrections. Our qualitative and quantitative experiments demonstrate that Context Canvas significantly enhances the capabilities of popular models such as Flux, Stable Diffusion, and DALL-E, and improves the functionality of ControlNet for fine-grained image editing tasks. To our knowledge, Context Canvas represents the first application of graph-based RAG in enhancing T2I models, representing a significant advancement for producing high-fidelity, context-aware multi-faceted images.

Introduction: The Long-Tail Challenge in AI Art

Current text-to-image (T2I) diffusion models are incredibly capable, but they suffer from a significant "long-tail" problem. Because they are trained on internet-scale datasets, they excel at common objects but often fail to accurately represent rare, culturally specific, or complex concepts. When faced with concepts like "geta sandals," "hair ice," or "red ginger," state-of-the-art models frequently hallucinate generic versions or fail to capture essential relational details.

Qualitative comparison showing how RAVEL (Context Canvas) corrects hallucinations in baseline models like Flux for rare concepts.

Existing Retrieval-Augmented Generation (RAG) methods typically rely on visual similarity, which is ineffective when high-quality visual exemplars for rare concepts are scarce. To address this, we present RAVEL (Rare Concept Generation and Editing via Automated Visual Enhancement and Logic), a training-free framework that leverages structured symbolic knowledge to guide image generation.

Methodology: The RAVEL Framework

Graph-Based RAG

Unlike traditional RAG systems that retrieve images, RAVEL queries a Knowledge Graph (KG) to extract structured data including attributes, taxonomies, and symbolic relationships. For instance, a prompt for "red ginger" triggers the retrieval of specific details regarding bract shape and color. This approach prioritizes symbolic structure over visual similarity, providing a reliable "blueprint" for the diffusion model even when its internal weights for a concept are weak.

Self-Correcting RAG-Guided Diffusion (SRD)

To ensure fidelity, we introduce the Self-Correcting RAG-Guided Diffusion (SRD) module. This iterative loop employs an LLM-based "critic" to compare generated outputs against retrieved KG attributes. If discrepancies are found (e.g., incorrect leaf shape), the module updates the prompt to emphasize missing features and regenerates the image until alignment is achieved.

The iterative SRD process refining the generation of "red ginger" to match botanical attributes.

Benchmarking and Results

To evaluate performance on rare concepts, we introduced three new benchmarks: MythoBench (mythological characters), Rare-Concept-1K (underrepresented categories), and NovelBench (archaic concepts from literature).

Our experiments demonstrate that RAVEL significantly outperforms baseline models and existing RAG-enhanced methods. Key findings include:

  • Accuracy Gains: RAVEL improved Attribute Accuracy by over 30% across all benchmarks compared to baseline models.
  • Human Preference: In user studies, participants preferred RAVEL-enhanced outputs over 90% of the time for models like Flux and DALL-E 3 when dealing with rare concepts.
  • Model Agnostic: As a prompt-level wrapper, RAVEL enhances various models (Flux, SDXL, DALL-E 3) without requiring retraining.

Conclusion

Context Canvas (RAVEL) represents a significant step forward in bridging the gap between common and rare concepts in AI art. By substituting unreliable visual retrieval with structured symbolic knowledge and an iterative self-correction loop, we enable diffusion models to generate high-fidelity, culturally accurate, and contextually rich images.

Summary updated

Abstract

We introduce a novel approach to enhance the capabilities of text-to-image models by incorporating a graph-based RAG. Our system dynamically retrieves detailed character information and relational data from the knowledge graph, enabling the generation of visually accurate and contextually rich images. This capability significantly improves upon the limitations of existing T2I models, which often struggle with the accurate depiction of complex or culturally specific subjects due to dataset constraints. Furthermore, we propose a novel self-correcting mechanism for text-to-image models to ensure consistency and fidelity in visual outputs, leveraging the rich context from the graph to guide corrections. Our qualitative and quantitative experiments demonstrate that Context Canvas significantly enhances the capabilities of popular models such as Flux, Stable Diffusion, and DALL-E, and improves the functionality of ControlNet for fine-grained image editing tasks. To our knowledge, Context Canvas represents the first application of graph-based RAG in enhancing T2I models, representing a significant advancement for producing high-fidelity, context-aware multi-faceted images.
BibTeX
@misc{venkatesh2024context,
  author = {Kavana Venkatesh and Yusuf Dalva and Ismini Lourentzou and Pinar Yanardag},
  title = {{Context Canvas: Enhancing Text-to-Image Diffusion Models with Knowledge Graph-Based RAG}},
  year = {2024},
  eprint = {2412.09614},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2412.09614},
}
Figure