Comparison of CoModGANs, LaMa and GLIDE for Art Inpainting Completing M.C Escher's Print Gallery

Lucia Cipolina-Kun, Simone Caenazzo, Gaston MazzeiWorkshops @ CVPR 2022 · DOI · arXiv

This study explores the intersection of high-level mathematics and generative AI to address a 60-year-old mystery in the art world: how to complete M.C. Escher’s famously unfinished masterpiece, Print Gallery (1956). By comparing three state-of-the-art AI models—CoModGANs, LaMa, and GLIDE—researchers evaluated which architecture could best navigate the complex recursive geometry of Escher's work.

The Challenge of the Droste Effect

Digital art restoration typically involves "inpainting," a technique used to fill missing or damaged regions of an image. However, Print Gallery presents a unique challenge due to its "Droste effect"—a recursive spiral where a man in a gallery views a print of a city that contains the very gallery he is standing in. Escher left the center of this spiral as a white void, as the extreme geometric warping made it manually impossible to complete.

M.C. Escher's Print Gallery (1956) featuring the incomplete central void.

Filling this void requires an AI to understand both large-scale context and extreme geometric distortion. Standard models often fail when the missing region is large or when the surrounding patterns follow non-Euclidean logic.

Methodology: A Three-Step Pipeline

To bridge the gap between standard AI inpainting and Escher's complex geometry, the researchers developed a specialized three-step pipeline:

  1. Mathematical Unrolling: Using conformal complex maps (exponential mappings), the twisted, spiral-shaped void was transformed into eight "straightened" Euclidean images. This converted the complex warping problem into a standard rectangular inpainting task.
  2. Model Comparison: The researchers tested three distinct architectures:
  3. CoModGANs: A GAN-based model designed for large-scale image completion.
  4. LaMa: A model utilizing Fast Fourier Convolutions, optimized for high-resolution pattern replication and texture consistency.
  5. GLIDE: A diffusion-based model that leverages text prompts to guide image generation with high semantic awareness.
  6. Rewarping: Once the AI filled the straightened images, an inverse mathematical map was applied to twist the results back into the original spiral structure.
The transformation process showing the unrolling and rewarping of the recursive gallery structure.

Key Insights and Model Performance

The study revealed significant trade-offs between the different AI architectures:

  • Pattern vs. Context: LaMa proved exceptional at repeating structural patterns like bricks and windows due to its Fourier-based approach. However, it lacked the "global" understanding required to invent new, semantically appropriate objects. Conversely, GLIDE’s diffusion process allowed for superior creativity and context awareness, though it was limited by lower output resolutions (256x256).
  • Multimodality: GLIDE’s ability to process text prompts (e.g., "a gallery with arches and wooden windows") enabled a "human-in-the-loop" approach, which is invaluable for historical art restoration where specific context is required.
  • Quantitative Results: Using "no-reference" metrics, GLIDE emerged as the winner in perceived quality, scoring highest on the Koniq scale (41.61) and lowest on BRISQUE (7.94), indicating fewer artifacts. LaMa won on edge sharpness (DOM score of 1.10), producing the crispest visual results.

Conclusion

The final reconstruction using GLIDE successfully captured the "infinite loop" of the lithograph, rotating the gallery by 157 degrees to align with Escher’s underlying mathematical intent. While diffusion models currently offer the best semantic results for complex art restoration, Fourier-based models like LaMa remain essential for high-resolution pattern fidelity. Future work may involve using Thin Plate Splines (TPS) to better align hand-drawn lines with AI-generated content, further blurring the line between human artistry and machine logic.

Summary updated

BibTeX
@inproceedings{cipolinakun2022comparison,
  author = {Lucia Cipolina-Kun and Simone Caenazzo and Gaston Mazzei},
  title = {{Comparison of CoModGANs, LaMa and GLIDE for Art Inpainting Completing M.C Escher's Print Gallery}},
  booktitle = {Workshops @ CVPR},
  year = {2022},
  doi = {10.1109/cvprw56347.2022.00087},
}
Figure