Analyzing Recursiveness in Multimodal Generative Artificial Intelligence: Stability or Divergence?

Javier Conde, Tobias Cheung, Gonzalo Martínez, Pedro Reviriego, Rik SarkararXiv 2024 · DOI · arXiv

Abstract

As generative artificial intelligence (AI) becomes a cornerstone of digital content creation, the internet is increasingly populated by synthetic data. This paper investigates the phenomenon of "Recursive Modality Changes" (RMC), where content is iteratively transformed between visual and textual modalities using state-of-the-art models like GPT-4o and DALL-E 3. Our findings indicate that these multimodal loops do not reach a stable equilibrium. Instead, they exhibit a consistent divergence, where original subjects are lost to semantic drift and stylistic "hallucinations," posing challenges for the long-term reliability of AI-generated ecosystems.

Introduction

The concept of "model collapse"—the degradation of AI performance when trained on its own output—is well-documented in unimodal contexts like text generation. However, the dynamics of multimodal recursion remain less understood. This study explores whether a "soul" or core identity of content survives when passed through a multimodal game of "Telephone." By repeatedly describing images and regenerating them from those descriptions, we test if the process converges on a stable "fixed point" or drifts into chaos.

Methodology: Recursive Modality Changes (RMC)

The researchers formalized a loop known as Recursive Modality Changes (RMC). The process begins with an initial image ($Image_0$) sourced from the COCO dataset. GPT-4o is tasked with generating a detailed caption ($Description_0$), which is then fed into DALL-E 3 to generate a subsequent image ($Image_1$). This cycle is repeated for 40 iterations.

The Recursive Modality Change (RMC) workflow, illustrating the iterative loop between image captioning and image generation.

To quantify the drift, the study utilized LPIPS for perceptual similarity, BERT and TF-IDF for semantic text similarity, and BLIP for image-text alignment. The experiments covered 60 images across categories such as living things, vehicles, and inanimate objects.

Results and Divergence

The primary finding is that divergence is the rule, not the exception. The AI loop fails to maintain a stable representation of the original input. For instance, an image of a person playing with a frisbee on a beach gradually transforms into a series of increasingly abstract, psychedelic cosmic mandalas.

A 40-generation trajectory showing the transformation of a person playing frisbee into a complex, abstract mandala.

Similarity Decay and Subject Loss

LPIPS similarity scores showed a sharp decline within the first 10 generations, eventually plateauing at a level indicating near-zero perceptual resemblance to the source. By the 39th generation, the original category (e.g., "person") was preserved in only 40% of the test cases. Interestingly, the study found that most information loss occurs during the text-to-image phase (DALL-E 3), where the model introduces creative flourishes that steer the trajectory away from the original intent.

Analysis of Stability Factors

Despite the overall trend of divergence, certain factors influenced the rate of change:

  • Stylistic Anchors: DALL-E 3’s "vivid" setting acted as a stabilizer compared to the "natural" setting. The high-contrast, dramatic style provided a consistent aesthetic framework that slowed semantic drift.
  • Category Persistence: Some objects proved more resilient. "Toilets" maintained their identity in 70% of generations, while "elephants" and "fire-hydrants" were frequently lost or transformed.
  • Cyclical Behavior: Trajectories often exhibited "mini-stable" cycles, remaining relatively consistent for approximately six generations before a significant "mutation" occurred, resetting the path of divergence.

Conclusion

The study demonstrates that multimodal AI loops are inherently unstable. The tendency for models to prioritize creative generation over faithful reconstruction leads to a cumulative loss of information. As AI-generated content continues to proliferate, understanding these divergence patterns is crucial for maintaining the integrity of digital information and preventing the "hallucinatory" drift of the web's visual landscape.

Summary updated

Abstract

One of the latest trends in generative Artificial Intelligence is tools that generate and analyze content in different modalities, such as text and images, and convert information from one to the other. From a conceptual point of view, it is interesting to study whether these modality changes incur information loss and to what extent. This is analogous to variants of the classical game telephone, where players alternate between describing images and creating drawings based on those descriptions leading to unexpected transformations of the original content. In the case of AI, modality changes can be applied recursively, starting from an image to extract a text that describes it; using the text to generate a second image, extracting a text that describes it, and so on. As this process is applied recursively, AI tools are generating content from one mode to use them to create content in another mode and so on. Ideally, the embeddings of all of them would remain close to those of the original content so that only small variations are observed in the generated content versus the original one. However, it may also be the case the distance to the original embeddings increases in each iteration leading to a divergence in the process and to content that is barely related to the original one. In this paper, we present the results of an empirical study on the impact of recursive modality changes using GPT-4o, a state-of-the-art AI multimodal tool, and DALL-E 3. The results show that the multimodality loop diverges from the initial image without converging to anything specific. We have observed differences depending on the type of initial image and the configuration of the models. These findings are particularly relevant due to the increasing use of these tools for content generation, reconstruction, and adaptation, and their potential implications for the content on the Internet of the future.
BibTeX
@misc{conde2024analyzing,
  author = {Javier Conde and Tobias Cheung and Gonzalo Martínez and Pedro Reviriego and Rik Sarkar},
  title = {{Analyzing Recursiveness in Multimodal Generative Artificial Intelligence: Stability or Divergence?}},
  year = {2024},
  eprint = {2409.16297},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2409.16297},
}
Figure