ObjectComposer: Consistent Generation of Multiple Objects Without Fine-tuning

Alec Helbling, Evan Montoya, Duen Horng ChauarXiv 2023 · arXiv · OpenReview

TL;DR

This paper introduces ObjectComposer, a novel, training-free method for generating images with multiple, consistently appearing objects based on user-provided reference images and a text prompt. It addresses the challenge of maintaining object identity across different scenes in text-to-image generation, a crucial step for applications like comic book creation.

What Problem Does This Paper Address?

Modern text-to-image generative models, while impressive, often struggle with a key limitation: consistency. If you ask the model to generate "a dog and a teapot" in various scenarios, the dog and teapot might look completely different each time. This lack of consistent object appearance hinders applications requiring stable characters or elements, such as generating comic strips with the same protagonist or creating consistent product mockups. Existing solutions often involve fine-tuning diffusion models for specific objects, which is computationally expensive and not scalable for real-time use or generating compositions of multiple distinct objects.

Methods

ObjectComposer tackles this by cleverly blending the outputs of existing, pre-trained diffusion models without any fine-tuning. The core idea is to leverage a model like BLIP-Diffusion, which can generate single objects based on a reference image, and combine its outputs with a background generation model.

Here's the breakdown:

  • Object-Specific Generation: For each object specified by a reference image and class (e.g., a specific dog image, a specific teapot image), ObjectComposer uses BLIP-Diffusion to generate its diffusion process independently.
  • Background Generation: Simultaneously, a separate diffusion model (like a vanilla Stable Diffusion) is used to generate the overall scene and background based on the text prompt.
  • Masked Blending: The key innovation lies in how these are combined. ObjectComposer uses masks (derived from cross-attention maps during a diffusion inversion process) to identify the spatial locations of each object. It then takes a weighted average of the diffusion latents from the object-specific and background models, guided by these masks. This ensures that the generated objects are placed correctly within the scene while retaining their reference appearance.

Datasets

Not discussed in the paper. The method relies on user-provided reference images for specific objects and general text prompts.

Key Insights and Ideas

  • Training-Free Consistency: The most significant insight is achieving consistent object generation without any fine-tuning. This dramatically reduces computational cost and makes the approach highly practical.
  • Leveraging Existing Strengths: ObjectComposer doesn't reinvent the wheel. It intelligently combines the strengths of specialized single-object generators (like BLIP-Diffusion) with general scene generators.
  • Masks as the Unifier: The use of attention-derived masks is crucial for spatially aligning and blending the different diffusion processes, ensuring a coherent final image.
  • Bridging the Gap for Downstream Tasks: This method directly addresses a critical bottleneck for creative applications that require consistent visual elements.

Key Results

The paper demonstrates ObjectComposer's ability to generate images where user-specified objects (like a particular dog and teapot) maintain their appearance across different prompts and scenes. Figure 1 visually contrasts this with vanilla Stable Diffusion, where the same objects appear differently in each generated image. The authors note that while the method is generally successful, there can be minor deviations in object appearance at times, but overall it shows strong potential for complex scene generation with multiple user-specified objects.

Limitations and Future Work

The authors acknowledge a limitation: at times, the appearance of an object can deviate slightly from the reference image. For instance, a generated teapot might have a similar color but a different shape.

Future work could explore:

  • Improving the fidelity of object appearance to the reference images.
  • Extending the method to handle even more complex scenes and a larger number of objects.
  • Investigating more sophisticated methods for mask generation and blending.

Summary updated

Abstract

Recent text-to-image generative models can generate high-fidelity images from text prompts. However, these models struggle to consistently generate the same objects in different contexts with the same appearance. Consistent object generation is important to many downstream tasks like generating comic book illustrations with consistent characters and setting. Numerous approaches attempt to solve this problem by extending the vocabulary of diffusion models through fine-tuning. However, even lightweight fine-tuning approaches can be prohibitively expensive to run at scale and in real-time. We introduce a method called ObjectComposer for generating compositions of multiple objects that resemble user-specified images. Our approach is training-free, leveraging the abilities of preexisting models. We build upon the recent BLIP-Diffusion model, which can generate images of single objects specified by reference images. ObjectComposer enables the consistent generation of compositions containing multiple specific objects simultaneously, all without modifying the weights of the underlying models.
BibTeX
@misc{helbling2023objectcomposer,
  author = {Alec Helbling and Evan Montoya and Duen Horng Chau},
  title = {{ObjectComposer: Consistent Generation of Multiple Objects Without Fine-tuning}},
  year = {2023},
  eprint = {2310.06968},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2310.06968},
}
Figure