MoMA: Multimodal LLM Adapter for Fast Personalized Image Generation

Bingchen Liu, Kunpeng Song

TL;DR

MoMA is a novel, tuning-free adapter for text-to-image diffusion models that enables fast, personalized image generation from a single reference image. It leverages multimodal LLMs to effectively blend subject identity with diverse contexts and textures, achieving state-of-the-art results without requiring per-instance fine-tuning.

What Problem Does This Paper Address?

The rapid advancement of text-to-image diffusion models has created a demand for robust image-to-image translation and personalization. While existing methods can generate impressive images from text, they often struggle with accurately preserving the identity of a specific subject while simultaneously adapting it to new contexts or textures. Many personalization techniques require computationally expensive fine-tuning for each new subject, limiting their practical application. Furthermore, achieving high detail fidelity, identity preservation, and prompt faithfulness simultaneously has been a persistent challenge.

Methods

MoMA addresses these challenges with a two-stage approach built upon a pre-trained multimodal LLM (specifically LLaVA-7B) and a diffusion model (Stable Diffusion v1.5).

  • Multimodal Generative Image-feature Decoder: This component uses the MLLM to extract visual features from a reference image and combine them with a target text prompt. It effectively generates "contextualized image features" that represent the subject in the desired new scenario. This is achieved by feeding the MLLM a specially crafted instruction sequence.
  • Self-Attention Feature Transfer with Masking: To further enhance detail fidelity, MoMA introduces a novel "self-attention shortcut." This mechanism efficiently transfers fine-grained image features from the reference image directly into the diffusion model's UNet. A crucial aspect is the masking procedure, which ensures that only foreground features are transferred, preventing interference from the reference image's background and improving the quality of generated backgrounds.
  • Tuning-Free and Plug-and-Play: A key innovation is that MoMA is entirely tuning-free. It acts as a plug-and-play module that can be directly integrated with existing diffusion models, requiring only a single reference image and a text prompt.

Datasets

  • Training: The model is trained on a dataset of 282K image/caption/image-mask triplets from the OpenImage-V7 dataset.
  • Evaluation:
    • DreamBooth dataset: Used for quantitative evaluation of subject fidelity.
  • Custom prompts and images: Used for qualitative evaluation across various subjects and prompts, including recontextualization and texture editing tasks.

Key Insights and Ideas

  • MLLMs as Generative Feature Extractors: The paper demonstrates that MLLMs can be effectively repurposed not just for understanding, but for generating rich, context-aware image features that are crucial for personalized image synthesis.
  • Self-Attention Shortcut for Detail: Introducing a direct "shortcut" for self-attention features significantly improves the fine-grained details of the generated subject, overcoming limitations of longer, more indirect feature propagation paths.
  • Masking for Background Integrity: The masking mechanism is critical for disentangling foreground and background information, leading to cleaner and more diverse backgrounds that are not negatively impacted by the subject's feature transfer.
  • Tuning-Free Personalization is Achievable: MoMA proves that high-quality, personalized image generation can be achieved without the need for per-instance fine-tuning, making it much more accessible and practical.

Key Results

  • Quantitative: MoMA consistently outperforms existing tuning-free personalization methods (IP-Adapter, BLIP-Diffusion) across various metrics, including DINO (subject fidelity), CLIP-T(c) (recontextualization), and CLIP-T(t) (texture editing). For example, in Table 1, MoMA achieves a CLIP-T(c) score of 0.803 and CLIP-T(t) of 0.335, surpassing baselines.
  • Qualitative:
    • MoMA excels at recontextualization, placing subjects in novel environments with high detail accuracy and favorable backgrounds (Fig. 4, Fig. 6, Fig. 7, Fig. 14-17).
  • It demonstrates strong texture editing capabilities, altering subject textures while preserving identity and shape (Fig. 5, Fig. 6, Fig. 18-19).
  • The model shows robustness across diverse subjects and prompts, including accessory incorporation and pose modification (Fig. 10).
  • It generalizes well to various community checkpoints fine-tuned from SD v1.5 (Fig. 11).

Limitations and Future Work

  • Text Rendering: The paper acknowledges that MoMA inherits limitations from Stable Diffusion v1.5 regarding text rendering accuracy, where text on objects (like bowls or cans) can be distorted or missing (Fig. 20).
  • Subject Difficulty: Some subjects are inherently harder to learn than others. While MoMA performs well on common subjects like dogs and cats, it can struggle with rarer subjects, especially when complex text prompts are involved (Fig. 20).
  • Deceptive Content: The authors recognize the potential for misuse in generating deceptive content and have intentionally excluded person-related subjects from their training data. They advise thorough examination of outputs before deployment.
  • Future Work: The paper suggests that the model's ability to generate convincing counterfeit images where individuals are central elements could be further explored, with a call for careful examination of outputs. The open-source nature of MoMA is intended to facilitate broader applications and further research.

Summary updated

Abstract

In this paper, we present MoMA: an open-vocabulary, training-free personalized image model that boasts flexible zero-shot capabilities. As foundational text-to-image models rapidly evolve, the demand for robust image-to-image translation grows. Addressing this need, MoMA specializes in subject-driven personalized image generation. Utilizing an open-source, Multimodal Large Language Model (MLLM), we train MoMA to serve a dual role as both a feature extractor and a generator. This approach effectively synergizes reference image and text prompt information to produce valuable image features, facilitating an image diffusion model. To better leverage the generated features, we further introduce a novel self-attention shortcut method that efficiently transfers image features to an image diffusion model, improving the resemblance of the target object in generated images. Remarkably, as a tuning-free plug-and-play module, our model requires only a single reference image and outperforms existing methods in generating images with high detail fidelity, enhanced identity-preservation and prompt faithfulness. Our work is open-source, thereby providing universal access to these advancements.
BibTeX
@inproceedings{liu2024moma,
  author = {Bingchen Liu and Kunpeng Song},
  title = {{MoMA: Multimodal LLM Adapter for Fast Personalized Image Generation}},
  booktitle = {European Conference on Computer Vision},
  year = {2024},
  doi = {10.1007/978-3-031-73661-2\_7},
}
Figure