POEM: Precise Object-level Editing via MLLM control

Marco Schouten, Mehmet Onurcan Kaya, Serge J. Belongie, Dim P. PapadopoulosarXiv 2025 · DOI · arXiv

Introduction

While diffusion models have revolutionized image generation, precise object-level editing remains a significant challenge. Current text-based methods often produce unintended global changes, while interaction-based methods are labor-intensive. POEM (Precise Object-level Editing via MLLM control) addresses this by leveraging Multimodal Large Language Models (MLLMs) to handle spatial reasoning and mathematical calculations, enabling high-precision modifications through simple text instructions.

Comparison of POEM with existing text-based and interaction-based editing methods.

Methodology

The core innovation of POEM is the decoupling of visual reasoning from the image drawing process. The pipeline consists of five distinct stages:

  1. Visual Grounding: An MLLM (such as QwenVL) identifies objects and their initial bounding boxes.
  2. Detection Refinement: Grounded-SAM is employed to generate high-fidelity segmentation masks, correcting the often imprecise coordinates provided by MLLMs.
  3. Edit Operation Parsing: A math-optimized LLM interprets the user's prompt (e.g., "move left by 120 px") and calculates a 3x3 transformation matrix.
  4. Transformation: The calculated matrix is applied to the object's mask to determine its target position.
  5. Edit Guided Image-to-Image Translation: A diffusion model performs the final rendering, using the masks to ensure the object is modified correctly while maintaining background consistency.
The POEM pipeline decouples reasoning steps (scene analysis and edit parsing) from drawing steps.

The VOCEdits Benchmark

To evaluate fine-grained object transformations, the researchers introduced VOCEdits, a benchmark built on PASCAL VOC 2012. It contains 3,030 samples featuring various affine transformations, including flipping, scaling, rotation, translation, and shearing. Unlike generic datasets, VOCEdits provides ground-truth transformation matrices and precise masks for rigorous quantitative evaluation.

Key Results and Insights

POEM consistently outperforms state-of-the-art models like InstructPix2Pix (IP2P) and LEDITS++:

  • Quantitative Performance: POEM achieved an average Intersection over Union (IoU) of 38.4%, surpassing LEDITS++ by 3%.
  • Operational Strengths: Significant gains were observed in translation (improving IoU from ~27% to ~32%) and scaling (improving IoU from ~33% to ~40%).
  • Reasoning vs. Drawing: The study highlights that diffusion models struggle with spatial "math." By delegating geometry to an LLM (specifically math-specialized models like QwenMath), the system achieves much higher reliability.

Limitations and Future Work

Despite its precision, POEM faces challenges with extreme transformations, such as scaling objects below 10% of their size or moving them to image boundaries where the diffusion model must "hallucinate" large areas of background. Additionally, the current framework is limited to rigid-body transformations and does not yet support non-rigid changes like altering human poses. Future work may integrate self-guidance signals to better preserve complex background textures.

Summary updated

BibTeX
@misc{schouten2025poem,
  author = {Marco Schouten and Mehmet Onurcan Kaya and Serge J. Belongie and Dim P. Papadopoulos},
  title = {{POEM: Precise Object-level Editing via MLLM control}},
  year = {2025},
  eprint = {2504.08111},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2504.08111},
}
Figure