Introduction
While diffusion models have revolutionized image generation, precise object-level editing remains a significant challenge. Current text-based methods often produce unintended global changes, while interaction-based methods are labor-intensive. POEM (Precise Object-level Editing via MLLM control) addresses this by leveraging Multimodal Large Language Models (MLLMs) to handle spatial reasoning and mathematical calculations, enabling high-precision modifications through simple text instructions.
Methodology
The core innovation of POEM is the decoupling of visual reasoning from the image drawing process. The pipeline consists of five distinct stages:
- Visual Grounding: An MLLM (such as QwenVL) identifies objects and their initial bounding boxes.
- Detection Refinement: Grounded-SAM is employed to generate high-fidelity segmentation masks, correcting the often imprecise coordinates provided by MLLMs.
- Edit Operation Parsing: A math-optimized LLM interprets the user's prompt (e.g., "move left by 120 px") and calculates a 3x3 transformation matrix.
- Transformation: The calculated matrix is applied to the object's mask to determine its target position.
- Edit Guided Image-to-Image Translation: A diffusion model performs the final rendering, using the masks to ensure the object is modified correctly while maintaining background consistency.
The VOCEdits Benchmark
To evaluate fine-grained object transformations, the researchers introduced VOCEdits, a benchmark built on PASCAL VOC 2012. It contains 3,030 samples featuring various affine transformations, including flipping, scaling, rotation, translation, and shearing. Unlike generic datasets, VOCEdits provides ground-truth transformation matrices and precise masks for rigorous quantitative evaluation.
Key Results and Insights
POEM consistently outperforms state-of-the-art models like InstructPix2Pix (IP2P) and LEDITS++:
- Quantitative Performance: POEM achieved an average Intersection over Union (IoU) of 38.4%, surpassing LEDITS++ by 3%.
- Operational Strengths: Significant gains were observed in translation (improving IoU from ~27% to ~32%) and scaling (improving IoU from ~33% to ~40%).
- Reasoning vs. Drawing: The study highlights that diffusion models struggle with spatial "math." By delegating geometry to an LLM (specifically math-specialized models like QwenMath), the system achieves much higher reliability.
Limitations and Future Work
Despite its precision, POEM faces challenges with extreme transformations, such as scaling objects below 10% of their size or moving them to image boundaries where the diffusion model must "hallucinate" large areas of background. Additionally, the current framework is limited to rigid-body transformations and does not yet support non-rigid changes like altering human poses. Future work may integrate self-guidance signals to better preserve complex background textures.
Summary updated