Introduction
Generative AI models, such as Stable Diffusion and Midjourney, have revolutionized content creation but remain fundamentally opaque. When these models generate an image, it is difficult to trace which specific training data influenced the output. This "black box" nature poses challenges for copyright enforcement, accountability, and transparency. Existing attribution methods typically rely on mathematical latent representations (vectors) that are efficient for computation but lack human interpretability. Furthermore, alternative methods requiring model retraining are computationally prohibitive.
Methodology: From Pixels to Structured Logic
To address these issues, we propose a framework that leverages Multimodal Large Language Models (MLLMs) and structured Knowledge Graphs (KGs) to provide transparent data attribution. The process moves from raw image data to human-readable semantic triples.
1. Ontology-Guided Extraction
Instead of generic descriptions, we utilize domain-specific ontologies (e.g., Fashionpedia) to guide an MLLM (such as LLaMA 3.2 Vision) in identifying specific attributes. This constraint forces the model to extract precise relationships like "silhouette," "material," and "pattern," significantly increasing the semantic density of the extracted data. Experiments showed that using an ontology increased the average number of triples per image from 3.4 to 7.9.
2. Knowledge Graph Construction
The extracted information is formatted as Subject-Predicate-Object triples and stored in a Neo4j graph database. Each image is represented as a star-shaped subgraph, where the central node is the image ID connected to various attribute nodes.
3. Semantic Comparison and Ranking
To determine the influence of training data, the system compares the KG of a generated image against the KGs of the training set. Similarity is ranked based on node overlap—the count of shared semantic attributes. This allows the system to explicitly show why two images are considered similar (e.g., both share a "V-neck" and "Cotton-blend Fabric").
Experimental Evaluation
The framework was tested using several datasets, including 44,000 fashion product images and frames from 10 iconic Studio Ghibli films.
Key Results
- Attribution Accuracy: In "unlearning" experiments—where influential images are identified, removed, and the model retrained—the KG-based method performed comparably to state-of-the-art embedding methods while providing human-readable explanations.
- Stylistic Fingerprinting: By isolating "style nodes" (e.g., specific lighting or color palettes) added during the generation process, the system successfully traced 97.4% of style-induced triples back to the original Ghibli movie corpus.
- Semantic Density: Retrieved training images formed coherent clusters sharing between 5 and 13 specific semantic attributes.
Discussion and Future Work
The core advantage of this approach is the shift from opaque embeddings to structured logic. While embeddings capture pixel-level similarity, Knowledge Graphs capture semantic intent. However, challenges remain regarding the initial computational cost of indexing large datasets and the dependence on high-quality, expert-designed ontologies.
Future work will focus on scaling graph database performance to handle billions of images and incorporating semantic inference to understand hierarchical relationships (e.g., recognizing that a "Trench Coat" is a type of "Outerwear" even if labels differ). By providing a clear audit trail of data influence, this framework offers a path toward more accountable and transparent generative AI.
Summary updated