Training Data Attribution for Image Generation using Ontology-Aligned Knowledge Graphs

Theodoros Aivalis, Iraklis A. Klampanos, Antonis Troumpoukis, Joemon M. JosearXiv 2025 · DOI · arXiv

Introduction

Generative AI models, such as Stable Diffusion and Midjourney, have revolutionized content creation but remain fundamentally opaque. When these models generate an image, it is difficult to trace which specific training data influenced the output. This "black box" nature poses challenges for copyright enforcement, accountability, and transparency. Existing attribution methods typically rely on mathematical latent representations (vectors) that are efficient for computation but lack human interpretability. Furthermore, alternative methods requiring model retraining are computationally prohibitive.

Methodology: From Pixels to Structured Logic

To address these issues, we propose a framework that leverages Multimodal Large Language Models (MLLMs) and structured Knowledge Graphs (KGs) to provide transparent data attribution. The process moves from raw image data to human-readable semantic triples.

The multi-step pipeline for image attribution, from image sets to ranked comparisons.

1. Ontology-Guided Extraction

Instead of generic descriptions, we utilize domain-specific ontologies (e.g., Fashionpedia) to guide an MLLM (such as LLaMA 3.2 Vision) in identifying specific attributes. This constraint forces the model to extract precise relationships like "silhouette," "material," and "pattern," significantly increasing the semantic density of the extracted data. Experiments showed that using an ontology increased the average number of triples per image from 3.4 to 7.9.

2. Knowledge Graph Construction

The extracted information is formatted as Subject-Predicate-Object triples and stored in a Neo4j graph database. Each image is represented as a star-shaped subgraph, where the central node is the image ID connected to various attribute nodes.

Example of an image mapped to semantic triples and its corresponding star-shaped knowledge graph.

3. Semantic Comparison and Ranking

To determine the influence of training data, the system compares the KG of a generated image against the KGs of the training set. Similarity is ranked based on node overlap—the count of shared semantic attributes. This allows the system to explicitly show why two images are considered similar (e.g., both share a "V-neck" and "Cotton-blend Fabric").

Experimental Evaluation

The framework was tested using several datasets, including 44,000 fashion product images and frames from 10 iconic Studio Ghibli films.

Key Results

  • Attribution Accuracy: In "unlearning" experiments—where influential images are identified, removed, and the model retrained—the KG-based method performed comparably to state-of-the-art embedding methods while providing human-readable explanations.
  • Stylistic Fingerprinting: By isolating "style nodes" (e.g., specific lighting or color palettes) added during the generation process, the system successfully traced 97.4% of style-induced triples back to the original Ghibli movie corpus.
  • Semantic Density: Retrieved training images formed coherent clusters sharing between 5 and 13 specific semantic attributes.

Discussion and Future Work

The core advantage of this approach is the shift from opaque embeddings to structured logic. While embeddings capture pixel-level similarity, Knowledge Graphs capture semantic intent. However, challenges remain regarding the initial computational cost of indexing large datasets and the dependence on high-quality, expert-designed ontologies.

Future work will focus on scaling graph database performance to handle billions of images and incorporating semantic inference to understand hierarchical relationships (e.g., recognizing that a "Trench Coat" is a type of "Outerwear" even if labels differ). By providing a clear audit trail of data influence, this framework offers a path toward more accountable and transparent generative AI.

Summary updated

Abstract

As generative models become powerful, concerns around transparency, accountability, and copyright violations have intensified. Understanding how specific training data contributes to a model's output is critical. We introduce a framework for interpreting generative outputs through the automatic construction of ontologyaligned knowledge graphs (KGs). While automatic KG construction from natural text has advanced, extracting structured and ontology-consistent representations from visual content remains challenging -- due to the richness and multi-object nature of images. Leveraging multimodal large language models (LLMs), our method extracts structured triples from images, aligned with a domain-specific ontology. By comparing the KGs of generated and training images, we can trace potential influences, enabling copyright analysis, dataset transparency, and interpretable AI. We validate our method through experiments on locally trained models via unlearning, and on large-scale models through a style-specific experiment. Our framework supports the development of AI systems that foster human collaboration, creativity and stimulate curiosity.
BibTeX
@misc{aivalis2025training,
  author = {Theodoros Aivalis and Iraklis A. Klampanos and Antonis Troumpoukis and Joemon M. Jose},
  title = {{Training Data Attribution for Image Generation using Ontology-Aligned Knowledge Graphs}},
  year = {2025},
  eprint = {2512.02713},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2512.02713},
}
Figure