Abstract
Image inpainting requires the synthesis of missing regions that are both semantically plausible and structurally coherent. Traditional methods often struggle with complex geometries, either ignoring structural constraints or relying on fragile two-stage pipelines. We present a multi-task learning framework that integrates structure prediction—specifically edges and gradients—directly into the image completion process. By employing a shared generator and explicit structure embedding, our approach minimizes error propagation and computational overhead. Evaluations on CelebA, Places2, and Facade datasets demonstrate that our method produces sharper, more realistic results than current state-of-the-art models.
Introduction
Image inpainting—the process of filling in missing or corrupted parts of an image—is a fundamental task in computer vision. A significant challenge in this field is "hallucinating" complex structures, such as facial features or architectural lines, in a way that remains consistent with the surrounding context.
The authors identify a major gap in existing deep learning approaches: most models either ignore structural knowledge entirely or use a "two-stage" process (e.g., first predicting edges, then filling pixels). These two-stage models are computationally expensive and prone to "error propagation," where a slightly wrong edge prediction in the first stage leads to a mangled final image. This paper seeks to integrate structural awareness directly into the learning process without the overhead or fragility of a disconnected two-step pipeline.
Methodology
The proposed framework leverages a unified architecture consisting of four primary components:
- Multi-Task Shared Generator: Instead of separate models, a single encoder-decoder network is trained to perform two tasks at once: completing the image and estimating the gradient/edge maps. This forces the model to learn a shared representation that understands how structure and color relate.
- Structure Embedding Layer: To ensure the structural information actually helps the inpainting, the model explicitly "embeds" the learned structure features back into the decoding phase. This acts as a set of "preconditions" or hints that guide the generation of fine details.
- Pyramid Structure Loss: The authors supervise the model at multiple scales using a novel loss function that compares predicted gradients to ground truth gradients, specifically using a "weighted edge mask" to force the model to pay extra attention to sharp boundaries.
- Attention Mechanism: To handle textures and repeating patterns, the model includes an attention layer (inspired by non-local means). This allows the model to "borrow" similar patches from distant, uncorrupted parts of the image to fill the hole.
Experimental Setup
The model was evaluated on three standard benchmarks:
- CelebA: A large-scale face attributes dataset focusing on complex, symmetrical structures.
- Places2: A dataset containing a wide variety of scenes and environments.
- Facade: A dataset of architectural buildings focusing on highly regular, geometric structures.
The authors tested the model using both regular (square) and irregular (randomly shaped) masks to simulate real-world damage.
Results and Analysis
The proposed method, referred to as "Ours," consistently outperformed state-of-the-art models like EdgeConnect (EG) and PEN-Net across both qualitative and quantitative measures.
Quantitative Performance
On the CelebA dataset with irregular masks, the model achieved an FID (Fréchet Inception Distance) of 1.227, significantly lower (better) than EG's 2.443. It also showed superior performance in VIF (Visual Information Fidelity), which correlates well with human perception.
Qualitative Comparison
Visual results show much sharper facial contours (eyes, mouths) and cleaner architectural lines compared to baselines, which often produced blurry or "melted" artifacts in complex regions.
Key Insights
- Gradients vs. Edges: While previous work focused on binary edge maps, this paper emphasizes gradient maps. Gradients contain more nuanced information about texture and high-frequency details, making them a better "skeleton" for the generator to flesh out.
- Implicit vs. Explicit Guidance: By training on both tasks simultaneously (multi-task), the model implicitly learns structure. By using the Embedding Layer, it explicitly uses that knowledge. The combination is more robust than either alone.
- Efficiency: Because it uses a shared generator rather than two separate networks, the model is more parameter-efficient and less likely to fail catastrophically if the initial structural guess is slightly off.
Conclusion
This paper demonstrates significant improvements in image inpainting by incorporating structural knowledge into a multi-task framework. While the results are promising, future work could investigate making the framework more architecture-agnostic and improving global semantic context for extremely large holes in high-resolution images. The authors plan to tailor this multi-task framework to more specific, specialized inpainting architectures.
Summary updated