Recursive InPainting (RIP): how much information is lost under recursive inferences?

Javier Conde, Miguel González, Gonzalo Martínez, Fernando Moral, Elena Merino-Gómez, Pedro ReviriegoarXiv 2024 · DOI · arXiv

As generative AI becomes a staple in creative workflows—think Photoshop’s "Generative Fill"—we are entering an era where digital content is no longer just created by AI, but modified by it repeatedly. A new paper titled "Recursive InPainting (RIP): how much information is lost under recursive inferences?" explores a digital version of the "photocopy of a photocopy" effect. The researchers investigate what happens when you apply AI inpainting to an image, then inpaint the result, and repeat the process until the original information dissolves.

TL;DR

Recursive InPainting (RIP) demonstrates that repeated AI-driven modifications lead to cumulative information loss and eventual image collapse. While realistic styles show more resilience, abstract or complex images quickly degenerate into non-meaningful artifacts, revealing the inherent biases and limitations of the underlying generative models.

What Problem Does This Paper Address?

We’ve heard about "model collapse" in the context of training—where AI models trained on AI-generated data eventually lose their grip on reality. This paper shifts the focus to the inference phase.

When a user recursively uses tools like inpainting to tweak an image, they are essentially running a feedback loop. The authors ask: Does this process stabilize, or does the image inevitably drift away from its original meaning? By quantifying this "perceptual drift," the study highlights the risks of information loss and misinformation that occur when AI-generated content is processed over and over again.

Methods

The researchers introduced a framework called Recursive InPainting (RIP). The process is straightforward but revealing:

  1. Start with an original image.
  2. Apply a random mask (a square of 64x64, 128x128, or 256x256 pixels) to remove a fragment.
  3. Reconstruct the missing fragment using Stable Diffusion 2 (fine-tuned for inpainting) without a text prompt, forcing the model to rely solely on surrounding visual context.
  4. Repeat the process for a predefined number of iterations.
Illustration of the Recursive InPainting (RIP) process

To measure the damage, they used LPIPS (Learned Perceptual Image Patch Similarity), a metric that uses deep neural networks (VGG, AlexNet, SqueezeNet) to mimic human judgment of image similarity.

Datasets

The study utilized three distinct sets of images to test the limits of the RIP process:

  • WikiArt: 100 randomly chosen art images to ensure a lack of selection bias.
  • COCO Dataset: 60 random real-world photographs to serve as a baseline for "realistic" reconstruction.
  • Curated Pictorial Styles: 80 images categorized into eight styles (e.g., Cubism, Realism, Pop Art, and Architectural Sketches) to see how different artistic "languages" affect AI stability.

Key Insights and Ideas

  • The "Anchoring" Effect of Reality: The AI performed significantly better on realistic styles (Realism, Historicism) and real photos. This suggests that models like Stable Diffusion are heavily biased toward the photographic data prevalent in their training sets.
  • Mask Size is a Catalyst for Collapse: Larger masks lead to faster degeneration. When the AI has less "original" context to anchor its reconstruction, it is more likely to hallucinate entirely new, unrelated elements.
  • Semantic Blindness: The AI doesn't "understand" the scene. In qualitative tests, the model often failed to comprehend perspective or face rotation, leading to "pixelated" or "alien" reconstructions that didn't fit the original composition.
  • Object Robustness: Large, geometrically simple objects (like trains or elephants) were more stable than small, complex ones (like apples or human faces). If a mask covers an entire small object, the AI has no "memory" of it and will replace it with something else or delete it entirely.
Example of RIP on a portrait: notice how the face changes over iterations

Key Results

  • Cumulative Drift: In all cases, LPIPS distance increased as the number of iterations grew. The slope of degradation was steepest for larger mask sizes.
  • Image Degeneration: In extreme cases, particularly with abstract styles like Expressionism, the image eventually collapsed into non-meaningful noise or completely different subjects. For example, a sketch of a lady by Van Gogh completely lost its subject after 16 iterations.
  • Qualitative "Hallucinations": The researchers noted that the AI often introduced new objects based on tiny leftover artifacts from the previous iteration, creating a "creative" rather than "reconstructive" result that moved further from the truth.
Van Gogh sketch degeneration over 16 iterations

Limitations and Future Work

The authors acknowledge several areas for further exploration:

  • Single Model Focus: The study only used Stable Diffusion. Other architectures, like GANs or newer Diffusion models, might exhibit different stability profiles.
  • Randomness vs. Intent: The use of random masks doesn't perfectly mirror how a human artist might use these tools. Future work could look at deterministic sequences or user-guided modifications.
  • Mitigation Strategies: A key takeaway is that smaller mask sizes reduce information loss. The authors suggest that developers could implement "degradation flags" to warn users when an image has drifted too far from its source.

Ultimately, RIP serves as a cautionary tale for the "generative loop." As AI-generated content begins to dominate the internet, understanding how these models degrade information through recursive use is vital for maintaining the integrity of our digital visual record.

Summary updated

Abstract

The rapid adoption of generative artificial intelligence (AI) is accelerating content creation and modification. For example, variations of a given content, be it text or images, can be created almost instantly and at a low cost. This will soon lead to the majority of text and images being created directly by AI models or by humans assisted by AI. This poses new risks; for example, AI-generated content may be used to train newer AI models and degrade their performance, or information may be lost in the transformations made by AI which could occur when the same content is processed over and over again by AI tools. An example of AI image modifications is inpainting in which an AI model completes missing fragments of an image. The incorporation of inpainting tools into photo editing programs promotes their adoption and encourages their recursive use to modify images. Inpainting can be applied recursively, starting from an image, removing some parts, applying inpainting to reconstruct the image, revising it, and then starting the inpainting process again on the reconstructed image, etc. This paper presents an empirical evaluation of recursive inpainting when using one of the most widely used image models: Stable Diffusion. The inpainting process is applied by randomly selecting a fragment of the image, reconstructing it, selecting another fragment, and repeating the process a predefined number of iterations. The images used in the experiments are taken from a publicly available art data set and correspond to different styles and historical periods. Additionally, photographs are also evaluated as a reference. The modified images are compared with the original ones by both using quantitative metrics and performing a qualitative analysis. The results show that recursive inpainting in some cases modifies the image so that it still resembles the original one while in others leads to degeneration.
BibTeX
@misc{conde2024recursive,
  author = {Javier Conde and Miguel González and Gonzalo Martínez and Fernando Moral and Elena Merino-Gómez and Pedro Reviriego},
  title = {{Recursive InPainting (RIP): how much information is lost under recursive inferences?}},
  year = {2024},
  eprint = {2407.09549},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2407.09549},
}
Figure