TL;DR
LumiVideo is an agentic system that automates video color grading by reasoning over industry-standard parameters rather than generating pixels. By combining Large Language Models (LLMs) with a "Tree of Thoughts" search and a human-in-the-loop reflection loop, it produces professional-grade cinematic visuals that are temporally consistent, fully interpretable, and editable in standard post-production software.
What Problem Does This Paper Address?
Professional color grading is a complex, iterative process that transforms "flat" log-encoded footage into emotionally resonant cinema. While AI has made strides in image editing, existing automated methods typically use "black-box" pixel-generation (e.g., diffusion models).
These approaches suffer from three fatal flaws in a professional context:
- Temporal Inconsistency: Generating pixels frame-by-frame often leads to flickering and spatial artifacts.
- Lack of Interpretability: Professionals cannot see why a model made a choice or adjust the underlying math.
- Workflow Disconnect: Generative models output rigid RGB arrays, whereas the industry relies on manipulable parameters like ASC-CDL (Color Decision List) and 3D LUTs (Look-Up Tables).
LumiVideo aims to bridge this gap by creating an agent that mimics the cognitive workflow of a human colorist.
Methods
The system decomposes color grading into four distinct stages:
- Perception: A dual-stream module analyzes the footage. The Physical Stream uses deterministic Color Space Transforms (CST) to map log footage to a normalized space and calculate objective exposure profiles. The Semantic Stream uses a Vision-Language Model (VLM) to identify the scene's mood, narrative context, and "protected tones" (like skin or sky) that must remain natural.
- Reasoning: This is the core engine. Instead of a single-step guess, the agent uses a Tree of Thoughts (ToT) strategy. It explores multiple grading hypotheses, guided by a Retrieval-Augmented Generation (RAG) framework filled with professional cinematic heuristics. A VLM "critic" evaluates low-resolution previews of these hypotheses to prune the tree and select the optimal parameter set.
- Execution: The agent compiles the chosen parameters into industry-standard ASC-CDL configurations and a globally consistent 3D LUT. To handle high-dynamic-range (HDR) content, the authors introduced an adaptive lift formulation and exponential highlight roll-off to prevent clipping and emulate analog film response.
- Reflection: An optional "human-in-the-loop" stage where a user can provide natural language feedback (e.g., "make the shadows slightly cooler"). The agent parses this intent, identifies the specific mathematical parameter to change, and updates the grade while locking all other parameters to ensure stability.
Datasets
The authors introduced LumiGrade, the first publicly available benchmark for automated grading from raw footage. It contains:
- Over 100 professionally shot, log-encoded video clips spanning various cameras (Sony, RED, ARRI, DJI, Nikon) and lighting conditions.
- Rich metadata for each clip (lens type, shooting location, time of day).
- A reference-graded subset of 40 clips processed by an experienced human colorist, providing both the final video and the exact ASC-CDL parameters used.
Key Insights and Ideas
- Parameters > Pixels: By outputting math (ASC-CDL) instead of pixels, LumiVideo guarantees 100% temporal consistency and allows the results to be imported directly into professional tools like DaVinci Resolve.
- The Power of ToT: The color parameter space is highly non-linear. The authors found that a standard "Chain of Thought" often gets stuck in suboptimal local decisions, whereas the Tree of Thoughts allows the agent to "audition" different looks before committing.
- Semantic Grounding: Explicitly identifying protected tone regions (e.g., skin tones between $15^\circ$ and $45^\circ$ hue) acts as a hard constraint, preventing the creative grading of the background from making people look unnatural.
- Linguistic Precision: The agent can map subjective modifiers like "slightly" vs. "heavily" to specific numerical step sizes (e.g., $\Delta l \approx 0.01$ vs. $0.05$), behaving like a reliable junior colorist.
Key Results
LumiVideo was compared against SOTA generative models (GPT-4, Gemini 3.1) and industry baselines.
- Quantitative: LumiVideo achieved the highest scores across technical quality (MANIQA, DeQA) and aesthetic metrics (CLIP-AS). In an "LLM-as-Judge" evaluation, it scored 7.90/10, actually surpassing the Human Expert's 7.80/10 in categories like Color Harmony and Tonal Range.
- Qualitative: As shown in Figure 3, generative models often introduced hallucinations or spatial artifacts, while LumiVideo maintained structural integrity.
- Human Evaluation: In a study with 48 participants, LumiVideo achieved a 38.2% win rate, closely trailing the Human Expert (43.6%) and far exceeding all other automated baselines. Professional colorists gave it a usability score of 4.4/5, noting its high predictability.
Limitations and Future Work
The primary limitation acknowledged by the authors is that the system currently applies a single global LUT to the entire frame. Professional colorists often use "Power Windows" or secondary qualifiers to adjust specific areas of a frame (e.g., brightening just a character's face).
Future work will focus on extending the agent's action space to include these spatially-varying adjustments, further closing the gap between AI automation and high-end manual artistry.
Summary updated