Anywhere: A Multi-Agent Framework for User-Guided, Reliable, and Diverse Foreground-Conditioned Image Generation

Tianyidan Xie, Rui Ma, Qian Wang, Xiaoqian Ye, Feixuan Liu, Ying Tai, Zhenyu Zhang, Lanjun Wang, Zili Yi

Introduction

Generating high-quality backgrounds for specific foreground objects—a task known as foreground-conditioned generation—is essential for applications in e-commerce, advertising, and augmented reality. While end-to-end inpainting models have made significant strides, they often struggle with maintaining the integrity of the original object or adhering strictly to complex user prompts. Anywhere is a novel multi-agent framework that addresses these challenges by replacing the traditional "black box" approach with a modular pipeline of specialized agents for semantic analysis, prompt engineering, and iterative refinement.

The Problem: Limitations of End-to-End Models

Existing end-to-end inpainting models frequently encounter four primary failure modes when generating backgrounds for a fixed foreground:

  1. Violated Object Integrity: Background pixels often "bleed" into the foreground object, or the model adds unwanted extensions to the original shape.
  2. Semantic Inconsistency: The generated scene may be contextually or physically implausible for the object (e.g., placing a kitchen appliance in a forest).
  3. Limited Diversity: Due to training biases, models often produce stereotypical and monotonous backgrounds.
  4. Textual Interference: Visual features of the object and user-provided text prompts often conflict, leading the model to ignore one or the other.
Comparison of the Anywhere framework against existing inpainting models, highlighting improvements in object integrity, semantic consistency, diversity, and textual adherence.

The Anywhere Framework

The core philosophy of Anywhere is to separate high-level semantic reasoning from low-level pixel synthesis. The framework consists of three main modules:

1. Prompt Generation Module

Instead of feeding a raw image directly to a generator, this module uses a Vision-Language Model (VLM) as a Foreground Analyzer to extract detailed attributes like shape, color, and pose. An LLM-based Prompt Creator then brainstorms creative background descriptions, which are filtered by a Prompt Selector to ensure semantic compatibility with the object.

2. Image Generation Module

This module utilizes a multi-step synthesis process:

  • Template Generator: Employs ControlNet (Canny edge-guided) to establish an initial background layout.
  • Template Repainter: A specialized agent that detects if the background has encroached on the foreground. It uses segmentation tools to compare the original mask with the generated one and selectively inpaints any violated areas to preserve object integrity.
  • Image Enhancer: A high-resolution refinement model (such as SDXL Refiner) harmonizes the final composite.

3. Quality Evaluator

A VLM-based critic assesses the final output for lighting consistency, color harmony, and structural coherence. If the quality falls below a certain threshold, the critic provides feedback to trigger a re-generation loop (capped at three iterations).

Experimental Results

The authors evaluated Anywhere using a curated dataset of 3,000 foreground objects sourced from LAION and MSCOCO, covering humans, vehicles, and household items. When compared against state-of-the-art models like BrushNet and HD-Painter, Anywhere demonstrated significant improvements:

  • Fidelity: Achieved a 4.6% improvement in Fréchet Inception Distance (FID).
  • Reliability: Reduced the occurrence of artifacts and integrity violations by 44%.
  • User Preference: Saw a 24% increase in user preference scores and a 33% boost in background diversity.
  • Text Adherence: Demonstrated a 5% increase in text-image matching scores in guided scenarios.

Conclusion and Future Work

By leveraging the "common sense" reasoning of LLMs and VLMs, the Anywhere framework effectively avoids the semantic hallucinations and integrity issues common in standard inpainting. While the multi-agent approach is more computationally intensive—requiring 2–3x more GPU time than end-to-end models—it provides a more reliable and diverse solution for high-stakes applications. Future work will focus on optimizing computational efficiency and improving performance on complex "corner cases," such as transparent or highly reflective objects.

Summary updated

Abstract

Recent advancements in image inpainting, particularly through diffusion modeling, have yielded promising outcomes. However, when tested in scenarios involving the completion of images based on the foreground objects, current methods that aim to inpaint an image in an end-to-end manner encounter challenges such as "over-imagination", inconsistency between foreground and background, and limited diversity. In response, we introduce Anywhere, a pioneering multi-agent framework designed to address these issues. Anywhere utilizes a sophisticated pipeline framework comprising various agents such as Visual Language Model (VLM), Large Language Model (LLM), and image generation models. This framework consists of three principal components: the prompt generation module, the image generation module, and the outcome analyzer. The prompt generation module conducts a semantic analysis of the input foreground image, leveraging VLM to predict relevant language descriptions and LLM to recommend optimal language prompts. In the image generation module, we employ a text-guided canny-to-image generation model to create a template image based on the edge map of the foreground image and language prompts, and an image refiner to produce the outcome by blending the input foreground and the template image. The outcome analyzer employs VLM to evaluate image content rationality, aesthetic score, and foreground-background relevance, triggering prompt and image regeneration as needed. Extensive experiments demonstrate that our Anywhere framework excels in foreground-conditioned image inpainting, mitigating "over-imagination", resolving foreground-background discrepancies, and enhancing diversity. It successfully elevates foreground-conditioned image inpainting to produce more reliable and diverse results.
BibTeX
@inproceedings{xie2025anywhere,
  author = {Tianyidan Xie and Rui Ma and Qian Wang and Xiaoqian Ye and Feixuan Liu and Ying Tai and Zhenyu Zhang and Lanjun Wang and Zili Yi},
  title = {{Anywhere: A Multi-Agent Framework for User-Guided, Reliable, and Diverse Foreground-Conditioned Image Generation}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.1609/aaai.v39i7.32797},
}
Figure