TL;DR
This paper introduces a novel "generator inversion" technique to attribute synthetic images to their specific generative model (GAN). By finding the latent input that best reconstructs a given image, they can determine which generator likely produced it, offering a more interpretable alternative to black-box attribution methods.
What Problem Does This Paper Address?
The rise of highly realistic synthetic imagery, particularly from Generative Adversarial Networks (GANs), presents a significant challenge for digital forensics. While traditional image forensics can identify the source camera, attributing an image to a specific synthetic generator is a newer, more complex problem. Existing methods for synthetic image attribution often act as "black boxes," providing a classification without explaining why or allowing for the reconstruction of the generation process. This paper aims to bridge this gap by developing a method that not only attributes an image to its generator but also recovers the latent input that produced it, offering a more transparent and functional approach.
Methods
The core of the paper's contribution lies in generator inversion. The authors frame the problem as finding a latent vector $z$ such that a generator $G$ produces an image $I_g$ that closely matches a given probe image $I$, i.e., $I_g = G(z) \approx I$. This is achieved by minimizing a loss function, typically the mean squared error between the probe image and the generated image.
For attribution, they consider a scenario with multiple known generators ($G_1, G_2, \dots, G_n$). For each generator, they independently perform inversion to find the latent vector that minimizes the reconstruction error for the probe image. The generator with the smallest reconstruction error is then assigned as the source.
To address the non-convex nature of the optimization and avoid local minima, they employ a multi-start optimization strategy, running the inversion process multiple times with different random initial guesses for the latent vector and selecting the best result. They also introduce a binary attribution score ($S$) for the case of two generators, which quantifies confidence in the attribution.
Datasets
- MNIST: Used for initial experiments with auto-encoders. The authors trained auto-encoders on MNIST digits, varying training data subsets and orderings to test attribution robustness.
- CelebA: Used for experiments with GANs. The authors utilized pre-trained ProGAN, SAGAN, and SNGAN generators trained on CelebA face images.
Key Insights and Ideas
- Inversion as Attribution: The fundamental insight is that the generator capable of best reconstructing a synthetic image is likely its true source. This is because generators are trained to map low-dimensional latent spaces to high-dimensional image spaces, and different training procedures lead to distinct mappings.
- Interpretability through Reconstruction: Unlike black-box methods, generator inversion provides a reconstructed image. This reconstruction serves as evidence for the attribution, and its quality can be visually assessed by an analyst.
- Robustness to Training Variations: The method demonstrates a surprising ability to distinguish between generators even when they are trained on very similar data (e.g., same digits but different random seeds or orderings), highlighting the subtle but detectable fingerprints left by the training process.
- Graceful Degradation with Compression: The method shows resilience to JPEG compression, a common real-world artifact. While performance degrades with lower quality factors, it remains significantly better than chance, indicating practical applicability.
Key Results
- High Attribution Accuracy on MNIST: The method achieved near-perfect attribution accuracy on MNIST digits, even when generators were trained with different random seeds or data orderings.
- Robustness to Compression: For JPEG compression, attribution performance remained significantly above chance even at a quality factor of 50.
- Strong Performance on CelebA GANs: Attribution accuracy reached 100% for SAGAN and SNGAN, and 95% for ProGAN on a dataset of 1500 synthetic face images.
- Visual Evidence: The paper provides numerous visual examples of successful reconstructions, demonstrating the interpretability of the method.
Limitations and Future Work
- Computational Complexity: Inverting a generator can be computationally expensive, especially with a large number of potential generators.
- Non-convex Optimization: While multi-start optimization helps, there's no guarantee of finding the global minimum reconstruction error.
- Black-Box Domain Extension: The current method requires access to the generator's weights (white-box). Future work aims to extend this to black-box scenarios, potentially by training proxy networks.
- Handling Different Output Ranges: The attribution score needs normalization when dealing with generators that produce outputs in different dynamic ranges.
- Training Neural Network Encoders: An alternative future direction is to train neural network encoders to directly map images to latent vectors, potentially speeding up the inversion process.
Summary updated