In-Context Brush: Zero-shot Customized Subject Insertion with Context-Aware Latent Space Manipulation

Yu Xu, Fan Tang, You Wu, Lin Gao, Oliver Deussen, Hongbin Yan, Jintao Li, Juan Cao, Tong-Yee Lee

Overview

In-Context Brush is a zero-shot framework designed to insert customized subjects into images with high fidelity and precise text control, requiring no model fine-tuning. By reformulating subject insertion as an "in-context learning" (ICL) task and applying dual-level latent space manipulation, it achieves state-of-the-art performance in identity preservation and prompt alignment.

The Challenge of Subject Insertion

Inserting a specific object, such as a unique handbag or a favorite plushie, into a new scene while adhering to a text prompt (e.g., "the bag is sitting on a park bench") is a significant challenge in image editing.

Subject insertion task: placing a specific handbag into a target scene.

Current methods typically fall into two categories:

  1. Tuning-based methods: These require fine-tuning on the specific subject, which is computationally expensive and often leads to "overfitting," reducing the model's ability to follow complex text instructions.
  2. Training-free methods: These utilize techniques like image inversion or blending. While faster, they often struggle to maintain the subject's identity or fail to align with textual prompts.

In-Context Brush addresses this gap by providing a fast, zero-shot method that preserves identity perfectly while remaining highly responsive to user text prompts.

Methodology

The framework is built upon a Multi-Modal Diffusion Transformer (MMDiT), specifically the Flux-1.0-fill model. The authors treat subject insertion as an In-Context Learning (ICL) problem.

Key technical innovations include:

  • In-Context Reformulation: The subject image and text prompt are concatenated as "demonstrations," while the target image (with a mask) serves as the "query." This leverages the model's pre-trained ability to identify relationships between examples.
  • Latent Feature Shifting (Intra-head): The authors mathematically demonstrate that ICL in attention blocks shifts query hidden states toward demonstration semantics. They explicitly amplify this shift to ensure subject features are injected into the target region.
  • Head-wise Reweighting (Inter-head): By identifying attention heads most activated by the text prompt and boosting them, the method improves the model's responsiveness to instructions like "make it stainless steel."
  • Token Blending: To maintain consistency with the background's style and lighting, original background latents are fused back into the process at each denoising step.

Evaluation and Results

The researchers evaluated In-Context Brush using the DreamBooth and COCO datasets, along with a robust internet collection, totaling 100 subject images and 130 scene images.

In-Context Brush outperformed eight state-of-the-art baselines, including Break-a-scene and TF-ICON, across several metrics:

  • Identity Preservation: Achieved top scores in DINO (0.7121) and CLIP-I (0.7957).
  • Image Quality: Reached the lowest FID score (122.61), indicating superior realism.
  • User Preference: In a study of 65 participants, the method was preferred ~66% of the time for identity alignment and ~73% for editing alignment.
Qualitative comparison of In-Context Brush in a virtual try-on application.

Beyond simple insertion, the paper demonstrates applications in Virtual Try-on, Compositional Generation, and Partly Insertion (e.g., swapping specific components of an object).

Limitations

A notable limitation occurs when the target background contains patterns similar to the subject (e.g., a car near a building with similar window patterns). In these cases, background features may "leak" onto the subject due to high similarity in the self-attention mechanism. Future work aims to introduce constraints to better disentangle subjects from similar background elements.

Summary updated

Abstract

Recent advances in diffusion models have enhanced multimodal-guided visual generation, enabling customized subject insertion that seamlessly "brushes" user-specified objects into a given image guided by textual prompts. However, existing methods often struggle to insert customized subjects with high fidelity and align results with the user's intent through textual prompts. In this work, we propose "In-Context Brush", a zero-shot framework for customized subject insertion by reformulating the task within the paradigm of in-context learning. Without loss of generality, we formulate the object image and the textual prompts as cross-modal demonstrations, and the target image with the masked region as the query. The goal is to inpaint the target image with the subject aligning textual prompts without model tuning. Building upon a pretrained MMDiT-based inpainting network, we perform test-time enhancement via dual-level latent space manipulation: intra-head "latent feature shifting" within each attention head that dynamically shifts attention outputs to reflect the desired subject semantics and inter-head "attention reweighting" across different heads that amplifies prompt controllability through differential attention prioritization. Extensive experiments and applications demonstrate that our approach achieves superior identity preservation, text alignment, and image quality compared to existing state-of-the-art methods, without requiring dedicated training or additional data collection.
BibTeX
@inproceedings{xu2025context,
  author = {Yu Xu and Fan Tang and You Wu and Lin Gao and Oliver Deussen and Hongbin Yan and Jintao Li and Juan Cao and Tong-Yee Lee},
  title = {{In-Context Brush: Zero-shot Customized Subject Insertion with Context-Aware Latent Space Manipulation}},
  booktitle = {SIGGRAPH Asia},
  year = {2025},
}
Figure