Agentic Planning with Reasoning for Image Styling via Offline RL

Subhojyoti Mukherjee, Stefano Petrangeli, Branislav Kveton, Trung Bui, Franck Dernoncourt, Arko Mukherjee

Abstract

Direct prompt-based editing often fails on complex transformations because vague and subjective prompts often require nuanced understanding of what should be changed in the image. Our core intuition is that leveraging compositional image editing tools rather than direct prompting profits from structured agent-level planning with explicit reasoning, leading to better results. This structured planning framework enables efficient offline RL post-training on quality-scored trajectories to improve performance. We present a tool-based agentic RL post-training framework that addresses this through structured planning with chain-of-thought reasoning. Our key contributions include: (1) A tool-based agentic planning methodology that combines a compositional library of orthogonal primitive transformations, structured context representation, and explicit per-step reasoning to decompose complex styling into interpretable tool sequences. (2) A synthetic data generation pipeline producing three large-scale datasets (each ~10K trajectories) with reasoning chains, plans, and quality scores. (3) Offline RL training methods for learning planners with reasoning, which consistently improve over the Edit-Only baseline in visual quality and instruction following. (4) Comprehensive evaluation across 4B and 8B parameter models showing that our methods outperform other baselines, including zero-shot GPT-4o, in the majority of compositional tasks.

Introduction

Direct prompt-based image editing (e.g., "Make this a golden-hour winter wonderland") often fails because natural language is inherently ambiguous. A single prompt doesn't specify the necessary sequence of changes—such as adjusting the time of day, adding snow effects, and modifying architectural styles—nor does it specify what should be preserved. Current state-of-the-art models often produce inconsistent results, misaligned colors, or structural artifacts when faced with these multi-dimensional transformations. We argue that the missing link is structured agent-level planning with explicit reasoning. Instead of a "one-shot" generation, the model should act as an agent that selects specific tools and explains why it is using them to reach the final aesthetic goal.

Methodology

We propose a Four-Stage Structured Editing Pipeline:

  1. Extract Structured Context: The model analyzes the input image across 10 dimensions (e.g., location, architecture, lighting, season) to create a text-based "state" representation.
  2. Plan Actions with Reasoning: Using a library of 10–20 orthogonal primitive tools (e.g., time_of_day, artistic_medium), the model generates a sequence of tool calls interleaved with Chain-of-Thought (CoT) reasoning.
  3. Synthesize Precise Instruction: The plan is converted into a highly detailed, unambiguous natural language instruction.
  4. Render Final Image: A frozen, black-box image editor executes the final instruction.

To train the planner, we utilize Offline Reinforcement Learning. We compare several methods: Reward-Weighted (RW), which weights training loss by quality scores; Standardized Reward-Weighted (SW), which normalizes rewards via z-scores for stability; and Direct Preference Optimization (DPO), a contrastive learning approach.

Experiments and Results

We generated three large-scale synthetic datasets (~10K trajectories each) to provide explicit tool-based styling supervision: Simple (atomic transformations), Regular (compositional transformations across 10 themes), and Complex (high-difficulty tasks with strict preservation constraints).

In our experiments, the Standardized Reward-Weighted (SW) method was a top performer, achieving the highest overall scores on regular tasks (78.77 on 4B, 77.86 on 8B). Qualitative results demonstrate that SW-trained models produce more coherent and visually pleasing images compared to the baseline.

A bedroom styled using the Standardized Reward-Weighted (SW) method, showing high coherence and visual quality.

Human validation confirmed the quality of the synthetic data with a 77% pass rate. Side-by-side comparisons showed that the RL-trained models consistently beat the baseline and direct-editing approaches. Notably, specialized 4B and 8B parameter models trained with these RL techniques can outperform general-purpose giants like GPT-4o on specific image-styling metrics.

Conclusion

This work demonstrates that structured planning with explicit reasoning significantly improves image styling performance. By leveraging offline RL and a tool-based agentic framework, we can transform vague prompts into precise, high-quality visual edits. Future work includes extending this framework to video editing and scaling to even larger tool libraries.

Summary updated

Abstract

Direct prompt-based editing often fails on complex transformations because vague and subjective prompts often require nuanced understanding of what should be changed in the image. Our core intuition is that leveraging compositional image editing tools rather than direct prompting profits from structured agent-level planning with explicit reasoning, leading to better results. This structured planning framework enables efficient offline RL post-training on quality-scored trajectories to improve performance. We present a tool-based agentic RL post-training framework that addresses this through structured planning with chain-of-thought reasoning. Our key contributions include: (1) A tool-based agentic planning methodology that combines a compositional library of orthogonal primitive transformations, structured context representation, and explicit per-step reasoning to decompose complex styling into interpretable tool sequences. (2) A synthetic data generation pipeline producing three large-scale datasets (each $\sim$10K trajectories) with reasoning chains, plans, and quality scores, as no existing datasets provide such supervision. Our datasets and code are publicly available at the HuggingFace repository. (3) Offline RL training methods for learning planners with reasoning as our core algorithmic contributions, which consistently improve over the Edit-Only baseline in visual quality and instruction following. (4) Comprehensive evaluation across 4B and 8B parameter Qwen3-VL models showing that our methods outperform other baselines in the majority of compositional tasks, validated by human evaluations.
BibTeX
@misc{mukherjee2026agentic,
  author = {Subhojyoti Mukherjee and Stefano Petrangeli and Branislav Kveton and Trung Bui and Franck Dernoncourt and Arko Mukherjee},
  title = {{Agentic Planning with Reasoning for Image Styling via Offline RL}},
  year = {2026},
  eprint = {2603.07148},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2603.07148},
}
Figure