LLM-I: LLMs are Naturally Interleaved Multimodal Creators

Zirun Guo, Feng Zhang, Kai Jia, Tao JinSubmission to ICLR 2026 · Rejected

TL;DR

Current models for interleaved text-and-image generation rely on a single visual synthesis engine, leaving them unable to provide factually accurate real-world photos or precise programmatic charts. LLM-I reframes interleaved generation as an agentic tool-use problem, empowering a central language model to dynamically orchestrate web search, diffusion synthesis, code execution, and image editing. Trained via reinforcement learning with a gated hybrid reward, LLM-I achieves state-of-the-art results across multiple benchmarks and scales effectively with test-time compute.

What Problem Does This Paper Address?

Generating documents with interleaved text and images (like tutorials, reports, and articles) requires both strong narrative coherence and context-appropriate visual assets. Today's dominant approaches fall into two camps:

  1. Two-stage pipelines: An LLM writes the narrative and calls a diffusion model (e.g., DALL-E or Stable Diffusion), which often causes cross-modal "semantic gaps" and restricts visual outputs to fixed, synthetic imagery.
  2. Unified, end-to-end models: Multimodal architectures (e.g., Show-o, Chameleon, Transfusion) synthesize text and tokens natively.

Both paradigms suffer from a "one-tool" bottleneck: they treat a generative diffusion/token-decoder as an "omniscient solver." While great for creative synthetic art, this architecture fails completely when an article requires factual grounding (e.g., an authentic photograph of the Eiffel Tower) or programmatic precision (e.g., an exact line plot generated from quarterly earnings data).

Methods

Rather than forcing a single model to synthesize every visual pixel from scratch, LLM-I mimics how human authors work—acting as an intelligent planner that delegates to specialized tools.

User Query ──> [ Central LLM/MLLM Planner ]
                     │  (Generates text with <imgen> tags)
                     ├──> Online Image Search (Google Search API)
                     ├──> Text-to-Image Diffusion (Seedream 3.0)
                     ├──> Code Execution Sandbox (Python / Matplotlib)
                     └──> Generative Image Editing (Seededit 3.0)

1. In-Context Tool Invocation

Within a single generative pass, the LLM emits a lightweight JSON-like tag whenever visual content is needed: <imgen>{"source": "<search|diffusion|code|edit>", "description": "...", "params": {...}}</imgen>. A parser executes the call, and the resulting image replaces the tag in the final document.

2. Reinforcement Learning Recipe

Models are trained with Group Relative Policy Optimization (GRPO) or Group Sequence Policy Optimization (GSPO for MoE architectures) using a hybrid reward function: $$R = w_{\text{rule}} R_{\text{rule}} + w_{\text{llm}} R_{\text{llm}} + w_{\text{mllm}} R_{\text{mllm}} R_{\text{rule}}$$ - Rule-based Reward ($R_{\text{rule}}$): Enforces JSON tag syntax and penalizes both under- and over-generation of images relative to implicit constraints. Crucially, $R_{\text{rule}}$ acts as a multiplicative gate on $R_{\text{mllm}}$, ensuring visual aesthetics are only rewarded if format constraints are satisfied. - LLM Judge ($R_{\text{llm}}$): Evaluates textual fluency, coherence, and tool-call placement/semantics. - MLLM Judge ($R_{\text{mllm}}$): Scores image quality, cross-modal alignment, and task relevance.

3. Test-Time Scaling (TTS)

During inference, LLM-I can trade compute for performance across a 4-stage pipeline:

  1. Sample $n$ candidate responses and discard malformed tool calls.
  2. Select the top-$k$ candidates with a judge model.
  3. Apply tool enhancements: run search and diffusion concurrently and pick the best via an MLLM; iteratively debug and re-run failed Python code.
  4. Polish the multimodal output with an MLLM before final selection.

Datasets

  • RL Training Dataset: ~4,000 samples constructed with Gemini 2.5 Pro and validated with GPT-4o. Prompts describe implicit goals and image constraints (disallowed, unconstrained, exact count, or $\ge 1$) without naming tools directly, forcing the policy to reason about tool selection.
  • Evaluated Benchmarks:
  • OpenING: 5,400 samples (text-only and multimodal inputs).
  • ISG Benchmark: >1,000 samples evaluating structural and block-level coherence.
  • LLMI-Bench (New): A 30-sample benchmark framed as complex "mini-projects" (e.g., scientific reports, financial analyses) that mandate informative charts and factual visuals. Unlike subjective scoring rubrics, it uses 10 concrete, sample-specific verifiable checks per task evaluated by GPT-4o alongside rigorous human Likert-scale review.

Key Insights and Ideas

  • Tool proficiency emerges naturally via RL: The authors never explicitly assigned a reward for "correct tool choice." Simply optimizing for multimodal coherence and constraint adherence drove tool-selection F1 score steadily upward during RL training.
  • The rule-based gate is vital: In ablation experiments, removing the rule-based constraint reward collapsed overall performance on OpenING from 8.18 down to 4.76, because the model learned to omit images entirely to avoid visual quality penalties.
  • Untrained models choke on too many tools: Base models without RL fine-tuning performed better when restricted to "search-only" than when exposed to the full toolkit, showing that naive prompting struggles with tool orchestration ambiguity. RL training resolved this, allowing LLM-I to leverage all tools synergistically.
  • Adaptive fallback: When diffusion models refused sensitive prompts (e.g., real public figures), the RL-trained agent automatically learned to pivot to web search.

Key Results

LLM-I was evaluated across four open-weight backbones (Qwen3-4B-Instruct, Qwen3-30B-Instruct, Qwen2.5-VL-7B, Qwen2.5-VL-32B):

Benchmark Best Baseline LLM-I (4B / 7B) LLM-I (30B / 32B)
OpenING (Overall) 8.20 (GPT-4o + DALL-E 3) 8.18 (LLM-I-4B) 8.66 (LLM-I-30B)
LLMI-Bench (Overall) 67.0 (GPT-4o w/ Tools) 82.3 (LLM-I-4B) 92.7 (LLM-I-30B)
LLMI-Bench (Tool Acc.) 67.9% (GPT-4o w/ Tools) 100.0% 100.0%
ISG (Holistic Score) 6.262 (ISG baseline) 8.413 (LLM-I-4B) 8.492 (LLM-I-30B)
  • Test-Time Scaling: Applying full TTS to LLM-I-4B pushed its LLMI-Bench rubric score from 88.9 to 95.1, outperforming the unscaled 30B model (94.8).

Limitations and Future Work

  • External Tool Output Quality: While tool invocation reached near 100% accuracy, rendering bugs still occur downstream (e.g., Matplotlib failing to format LaTeX/subscripts like $\text{CO}_2$ properly).
  • Latency of Test-Time Scaling: While tool calls execute in parallel, the candidate polishing phase requires rewriting long responses, adding ~16–20 seconds of inference latency per iteration.
  • Benchmark Size: LLMI-Bench comprises 30 deep "mini-projects." While designed to make rigorous human evaluation tractable, expanding to a broader automated suite remains future work.

Summary updated

Abstract

We propose LLM-Interleaved (**LLM-I**), a flexible and dynamic framework that reframes interleaved image-text generation as a tool-use problem. LLM-I is designed to overcome the "one-tool" bottleneck of current unified models, which are limited to synthetic imagery and struggle with tasks requiring factual grounding or programmatic precision. Our framework empowers a central LLM or MLLM agent to intelligently orchestrate a diverse toolkit of specialized visual tools, including online image search, diffusion-based generation, code execution, and image editing. The agent is trained to select and apply these tools proficiently via a Reinforcement Learning (RL) framework that features a hybrid reward system combining rule-based logic with judgments from LLM and MLLM evaluators. Trained on a diverse new dataset using four different model backbones, LLM-I demonstrates state-of-the-art performance, outperforming existing methods by a large margin across four benchmarks. We also introduce a novel test-time scaling strategy that provides further performance gains.
BibTeX
@misc{guo2025llm,
  author = {Zirun Guo and Feng Zhang and Kai Jia and Tao Jin},
  title = {{LLM-I: LLMs are Naturally Interleaved Multimodal Creators}},
  year = {2025},
  eprint = {2509.13642},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2509.13642},
}

Loading related content...

Figure