Image Inpainting

Marcelo Bertalmı́o, Guillermo Sapiro, Vicent Caselles, Coloma Ballester2005 · DOI

TL;DR

This paper introduces a novel digital image inpainting algorithm that mimics professional art restoration techniques. By automatically propagating information from surrounding regions along isophote lines, it effectively fills in damaged or removed areas of an image, offering a significant improvement over existing methods that require manual intervention or are limited to specific image types.

What Problem Does This Paper Address?

The paper tackles the challenge of "inpainting," which is the art of seamlessly modifying an image to fill in missing or damaged regions. This has applications ranging from restoring ancient paintings and photographs to removing unwanted objects or text from digital images. Traditional methods either struggle with large, complex regions, rely on manual user input for every detail, or are limited to specific types of images like film sequences. The authors aim to bridge this gap by developing an algorithm that is automatic, handles diverse image structures, and replicates the intuitive approach of human restorers.

Methods

The core of the proposed algorithm lies in its ability to automatically propagate information from the image's surroundings into the region to be inpainted. This is achieved by:

  • Isophote Propagation: The algorithm identifies "isophote lines" (lines of constant intensity) at the boundary of the region to be inpainted and extends them inwards. This ensures that the newly filled-in areas maintain the local structure and flow of the original image.
  • Anisotropic Diffusion: To prevent the propagated lines from crossing and to maintain image sharpness, the algorithm incorporates anisotropic diffusion. This process smooths the image in a direction-dependent manner, preserving edges and important features.
  • Iterative Refinement: The inpainting process is iterative. In each step, the algorithm estimates the image's smoothness and projects this variation along the isophote direction to update the pixels within the inpainted region. This is repeated until a steady state is reached.
  • Multiresolution Approach: For large regions, a multiresolution strategy is employed to speed up the computation, making the process more efficient.
  • Color Image Handling: Color images are treated as three separate planes (similar to the LUV color model), with the inpainting algorithm applied independently to each.

What's novel is the automatic nature of the information propagation and the handling of complex, varied surrounding backgrounds without explicit user guidance beyond marking the region to be filled.

Datasets

Not discussed in the paper. The authors mention using images available from public databases over the Internet for their examples.

Key Insights and Ideas

  • Mimicking Human Restoration: The algorithm's success stems from its inspiration by how professional restorers work – by continuing existing lines and structures rather than guessing or applying generic textures.
  • Isophote Continuity is Key: The core idea of propagating information along isophote lines is crucial for achieving visually plausible results, especially in structured regions.
  • Automaticity is a Game-Changer: Eliminating the need for users to specify what information to use for filling significantly broadens the applicability of inpainting.
  • Topology is Not a Barrier: The algorithm is designed to handle regions of arbitrary shape and complexity, a significant advantage over methods that require simple topologies.
  • Anisotropic Diffusion for Sharpness: The integration of anisotropic diffusion is vital for maintaining image quality and preventing artifacts that could arise from simple smoothing.

Key Results

The paper demonstrates the algorithm's effectiveness through several examples:

  • Restoration of Damaged Photographs: The algorithm successfully restores cracks and missing parts in black and white photographs, preserving the original image's structure.
  • Removal of Superimposed Text: Text placed on images is effectively removed, leaving behind a seamless reconstruction of the underlying image.
  • Removal of Objects: Objects like microphones or wires are removed from images, showcasing the algorithm's ability to handle complex scene modifications.
  • Speed: For color images, the inpainting process completed in under 5 minutes using non-optimized C++ code on a Pentium II PC. A 2-level multiresolution scheme reduced this to 2 minutes for some examples.
  • Robustness: The algorithm shows robustness to variations in the initial conditions of the region to be inpainted, consistently producing similar results.

Limitations and Future Work

  • Texture Reproduction: The authors acknowledge that the algorithm struggles with reproducing large, complex textured regions (as shown in Figure 14). This is a primary area for future work, with plans to combine their approach with texture synthesis techniques.
  • High-Order PDEs: The current approach, while effective, is conceptually related to high-order partial differential equations (PDEs). The authors suggest investigating lower-order PDEs for a more formal mathematical understanding and potentially improved results.
  • Formal Guarantees: While experimental results are strong, the authors aim to develop formal mathematical proofs for the uniqueness and existence of solutions, which would provide stronger theoretical backing.
  • Coupled Variational Formulations: Future work includes coupling the recovery of isophote directions and gray values using variational formulations to simultaneously solve for both, potentially leading to more robust and theoretically grounded solutions.

Summary updated

BibTeX
@misc{bertalmo2005image,
  author = {Marcelo Bertalmı́o and Guillermo Sapiro and Vicent Caselles and Coloma Ballester},
  title = {{Image Inpainting}},
  year = {2005},
  doi = {10.21236/ada437378},
}
Figure