LumiVideo: An Intelligent Agentic System for Video Color Grading

Yuchen Guo, Junli Gong, Hongmin Cai, Yiu-ming Cheung, Weifeng SuarXiv 2026 · arXiv

TL;DR

LumiVideo is an agentic system that automates video color grading by reasoning over industry-standard parameters rather than generating pixels. By combining Large Language Models (LLMs) with a "Tree of Thoughts" search and a human-in-the-loop reflection loop, it produces professional-grade cinematic visuals that are temporally consistent, fully interpretable, and editable in standard post-production software.

What Problem Does This Paper Address?

Professional color grading is a complex, iterative process that transforms "flat" log-encoded footage into emotionally resonant cinema. While AI has made strides in image editing, existing automated methods typically use "black-box" pixel-generation (e.g., diffusion models).

These approaches suffer from three fatal flaws in a professional context:

  1. Temporal Inconsistency: Generating pixels frame-by-frame often leads to flickering and spatial artifacts.
  2. Lack of Interpretability: Professionals cannot see why a model made a choice or adjust the underlying math.
  3. Workflow Disconnect: Generative models output rigid RGB arrays, whereas the industry relies on manipulable parameters like ASC-CDL (Color Decision List) and 3D LUTs (Look-Up Tables).

LumiVideo aims to bridge this gap by creating an agent that mimics the cognitive workflow of a human colorist.

Methods

The system decomposes color grading into four distinct stages:

  1. Perception: A dual-stream module analyzes the footage. The Physical Stream uses deterministic Color Space Transforms (CST) to map log footage to a normalized space and calculate objective exposure profiles. The Semantic Stream uses a Vision-Language Model (VLM) to identify the scene's mood, narrative context, and "protected tones" (like skin or sky) that must remain natural.
  2. Reasoning: This is the core engine. Instead of a single-step guess, the agent uses a Tree of Thoughts (ToT) strategy. It explores multiple grading hypotheses, guided by a Retrieval-Augmented Generation (RAG) framework filled with professional cinematic heuristics. A VLM "critic" evaluates low-resolution previews of these hypotheses to prune the tree and select the optimal parameter set.
  3. Execution: The agent compiles the chosen parameters into industry-standard ASC-CDL configurations and a globally consistent 3D LUT. To handle high-dynamic-range (HDR) content, the authors introduced an adaptive lift formulation and exponential highlight roll-off to prevent clipping and emulate analog film response.
  4. Reflection: An optional "human-in-the-loop" stage where a user can provide natural language feedback (e.g., "make the shadows slightly cooler"). The agent parses this intent, identifies the specific mathematical parameter to change, and updates the grade while locking all other parameters to ensure stability.
Fig. 2. Architecture of LumiVideo. As an example, we set up max iteration times to 5 here.

Datasets

The authors introduced LumiGrade, the first publicly available benchmark for automated grading from raw footage. It contains:

  • Over 100 professionally shot, log-encoded video clips spanning various cameras (Sony, RED, ARRI, DJI, Nikon) and lighting conditions.
  • Rich metadata for each clip (lens type, shooting location, time of day).
  • A reference-graded subset of 40 clips processed by an experienced human colorist, providing both the final video and the exact ASC-CDL parameters used.

Key Insights and Ideas

  • Parameters > Pixels: By outputting math (ASC-CDL) instead of pixels, LumiVideo guarantees 100% temporal consistency and allows the results to be imported directly into professional tools like DaVinci Resolve.
  • The Power of ToT: The color parameter space is highly non-linear. The authors found that a standard "Chain of Thought" often gets stuck in suboptimal local decisions, whereas the Tree of Thoughts allows the agent to "audition" different looks before committing.
  • Semantic Grounding: Explicitly identifying protected tone regions (e.g., skin tones between $15^\circ$ and $45^\circ$ hue) acts as a hard constraint, preventing the creative grading of the background from making people look unnatural.
  • Linguistic Precision: The agent can map subjective modifiers like "slightly" vs. "heavily" to specific numerical step sizes (e.g., $\Delta l \approx 0.01$ vs. $0.05$), behaving like a reliable junior colorist.

Key Results

LumiVideo was compared against SOTA generative models (GPT-4, Gemini 3.1) and industry baselines.

  • Quantitative: LumiVideo achieved the highest scores across technical quality (MANIQA, DeQA) and aesthetic metrics (CLIP-AS). In an "LLM-as-Judge" evaluation, it scored 7.90/10, actually surpassing the Human Expert's 7.80/10 in categories like Color Harmony and Tonal Range.
  • Qualitative: As shown in Figure 3, generative models often introduced hallucinations or spatial artifacts, while LumiVideo maintained structural integrity.
  • Human Evaluation: In a study with 48 participants, LumiVideo achieved a 38.2% win rate, closely trailing the Human Expert (43.6%) and far exceeding all other automated baselines. Professional colorists gave it a usability score of 4.4/5, noting its high predictability.
Fig. 3. Qualitative comparison with other SOTA methods.

Limitations and Future Work

The primary limitation acknowledged by the authors is that the system currently applies a single global LUT to the entire frame. Professional colorists often use "Power Windows" or secondary qualifiers to adjust specific areas of a frame (e.g., brightening just a character's face).

Future work will focus on extending the agent's action space to include these spatially-varying adjustments, further closing the gap between AI automation and high-end manual artistry.

Summary updated

Abstract

Video color grading is a critical post-production process that transforms flat, log-encoded raw footage into emotionally resonant cinematic visuals. Existing automated methods act as static, black-box executors that directly output edited pixels, lacking both interpretability and the iterative control required by professionals. We introduce LumiVideo, an agentic system that mimics the cognitive workflow of professional colorists through four stages: Perception, Reasoning, Execution, and Reflection. Given only raw log video, LumiVideo autonomously produces a cinematic base grade by analyzing the scene's physical lighting and semantic content. Its Reasoning engine synergizes an LLM's internalized cinematic knowledge with a Retrieval-Augmented Generation (RAG) framework via a Tree of Thoughts (ToT) search to navigate the non-linear color parameter space. Rather than generating pixels, the system compiles the deduced parameters into industry-standard ASC-CDL configurations and a globally consistent 3D LUT, analytically guaranteeing temporal consistency. An optional Reflection loop then allows creators to refine the result via natural language feedback. We further introduce LumiGrade, the first log-encoded video benchmark for evaluating automated grading. Experiments show that LumiVideo approaches human expert quality in fully automatic mode while enabling precise iterative control when directed.
BibTeX
@misc{guo2026lumivideo,
  author = {Yuchen Guo and Junli Gong and Hongmin Cai and Yiu-ming Cheung and Weifeng Su},
  title = {{LumiVideo: An Intelligent Agentic System for Video Color Grading}},
  year = {2026},
  eprint = {2604.02409},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2604.02409},
}
Figure