I Dream My Painting: Connecting MLLMs and Diffusion Models via Prompt Generation for Text-Guided Multi-Mask Inpainting

Nicola Fanelli, Gennaro Vessio, Giovanna CastellanoWACV 2025

TL;DR

This paper introduces "multi-mask inpainting," a novel task where multiple regions of an image are inpainted simultaneously, each guided by a unique text prompt. They propose a pipeline that uses multimodal large language models (MLLMs) to automatically generate these prompts from masked images and then employs a modified diffusion model (Stable Diffusion) with rectified cross-attention to perform the multi-mask inpainting. This approach automates a complex creative task, enabling more nuanced and controlled image editing.

What Problem Does This Paper Address?

Traditional image inpainting aims to fill missing or corrupted regions in an image to seamlessly blend with the surroundings. While text-guided diffusion models like Stable Diffusion have revolutionized inpainting by allowing users to specify desired content via text prompts, they typically handle one prompt per inpainting operation. This paper addresses the limitation of single-prompt inpainting by introducing multi-mask inpainting, where an image can have multiple distinct masked regions, each requiring its own specific textual guidance. The challenge lies not only in generating multiple relevant prompts for these regions but also in ensuring the diffusion model correctly associates each prompt with its corresponding masked area, preventing concept blending and ensuring accurate spatial adherence. Furthermore, the authors aim to automate the entire process, from prompt generation to inpainting, making it accessible even without manual annotation.

Methods

The proposed pipeline consists of two main stages: prompt generation and multi-mask inpainting.

  1. Automatic Data Annotation: To overcome the lack of object-level annotations for artworks, they developed a pipeline using MLLMs. First, Kosmos-2 is used to identify objects and generate bounding boxes. Then, cropped images of these objects are fed to LLaVA, along with contextual text tokens, to generate detailed object-level captions.
  2. Prompt Generation: This stage leverages a fine-tuned multimodal large language model (MLLM), specifically LLaVA. Given a masked image and the generated object-level descriptions, the MLLM is trained to predict textual prompts for each masked region. This is framed as a visual instruction-following task. They use QLoRA for efficient fine-tuning. The generated prompts are formatted with special tags corresponding to the colors assigned to each mask to maintain spatial correspondence.
  3. Multi-Mask Inpainting: The generated prompts are then fed into Stable Diffusion. To ensure each prompt guides its specific masked region, they adapt the rectified cross-attention (RCA) technique from FreestyleNet. Standard cross-attention in diffusion models can lead to prompt blending when multiple prompts are concatenated. RCA modifies the attention mechanism by downsampling a spatial layout derived from the masks and using it to rectify the attention map. This makes prompt tokens more influential for their corresponding regions. They also introduce modifications to RCA to handle overlapping masks and regions that are not masked.

Datasets

  • WikiArt: A dataset of digitized artistic paintings. The authors collected images from this source and developed an automatic annotation pipeline to generate bounding boxes and object-level descriptions for their experiments. They retained 102,276 images after filtering based on mask area constraints.
  • Densely Captioned Images (DCI) dataset: A dataset of photographic scenes with region-level annotations. This dataset was used to demonstrate the pipeline's adaptability beyond artistic images.

Key Insights and Ideas

  • Automating Prompt Generation for Multi-Mask Inpainting: The core innovation is the ability to automatically generate multiple, distinct prompts for different masked regions within a single image, eliminating the need for manual prompt engineering for each mask.
  • Rectified Cross-Attention (RCA) for Spatial Control: Adapting RCA from layout-to-image synthesis proves crucial for enforcing prompt-region correspondence in diffusion models, preventing concept blending and ensuring accurate inpainting.
  • MLLMs as Powerful Prompt Generators: Fine-tuning MLLMs like LLaVA on masked images and object descriptions allows them to predict creative and contextually relevant inpainting prompts, demonstrating their potential beyond simple captioning.
  • Artworks as a Challenging Testbed: Using digitized paintings highlights the pipeline's robustness in handling complex imagery and abstract concepts, where object-level annotations are scarce and prompt generation is inherently more difficult.

Key Results

  • Prompt Generation Quality: Their fine-tuned LLaVA-MultiMask model achieved competitive prompt generation quality, outperforming the base LLaVA model and a single-mask fine-tuned variant on metrics like accuracy and BLEU, especially in multi-mask scenarios.
  • Inpainting Performance: The proposed pipeline, using Stable Diffusion with RCA and generated prompts, demonstrated strong prompt-following capabilities, significantly outperforming baselines that did not use RCA or were not fine-tuned for the task. While not always achieving the absolute best FID or LPIPS scores (which can be less sensitive to prompt adherence), it excelled in metrics like CLIPSim-T2I, indicating better alignment between the inpainted content and its intended prompt.
  • Temperature Tuning for Diversity: Adjusting the sampling temperature during prompt generation offered a trade-off between prompt plausibility (CLIPSim) and diversity (Distinct-N, Self-BLEU), suggesting that intermediate temperatures can yield useful and varied prompt suggestions.

Limitations and Future Work

  • Prompt Repetition and Overlap: The prompt generator sometimes produces repetitive concepts for overlapping masks or overly long prompts. Improving this might involve more advanced NLP techniques for prompt preprocessing or postprocessing.
  • Crowded Areas and Tiny Masks: Multi-mask inpainting still struggles in areas with very small or overlapping masks, where prompts might be ignored or the model may fail to capture fine details.
  • Qualitative Evaluation: While quantitative metrics are provided, the authors acknowledge the need for more qualitative analysis, especially in complex scenarios.
  • Future Work: The authors suggest exploring NLP techniques for better prompt generation and investigating more advanced methods to handle challenging inpainting scenarios with dense masks.

Summary updated

BibTeX
@inproceedings{fanelli2025i,
  author = {Nicola Fanelli and Gennaro Vessio and Giovanna Castellano},
  title = {{I Dream My Painting: Connecting MLLMs and Diffusion Models via Prompt Generation for Text-Guided Multi-Mask Inpainting}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2025},
}
Figure