TL;DR
AVA-Encoder introduces an auto-encoding framework that bridges the gap between raw video and AI creative agents by translating films into a structured, text-based "Film Knowledge Graph" and back. Driven by an innovative dual-loop textual-gradient optimization process, it dramatically improves video reconstruction fidelity and enables precise, topology-aware video editing.
What Problem Does This Paper Address?
While foundation models have made massive strides in generating video content, AI video creation agents still struggle to produce truly cinematic-grade films. A core reason for this is a fundamental mismatch between video space and agent space.
Cinematic videos are dense, highly coupled multimodal streams where stories, character actions, camera movements, and audio are tightly intertwined across space and time. Conversely, AI agents learn and operate best within structured representations like text, code, and graphs. Existing video representations fail to bridge this gap: low-level visual tokens (pixels or latents) are uninterpretable for an agent to reason over, while standard textual captions or sparse scene graphs discard the fine-grained visual and production details required to actually recreate a scene.
Without a representation that is simultaneously agent-readable, editable, and cinematically faithful, agents cannot effectively study and learn filmmaking knowledge from high-quality human films.
Methods
To solve this, the authors propose the Agentic Video Auto-Encoder (AVA-Encoder). The framework operates like a classic auto-encoder but uses a structured text graph as its bottleneck and an LLM-driven optimization process instead of traditional backpropagation.
The system consists of three main components:
- Hierarchical Agentic Video Encoder: This module breaks down an input film into three granularities: film-level, shot-level, and keyframe-level. Crucially, it uses inter-level context injection, passing high-level narrative and entity registries down to finer stages to ensure global consistency (e.g., keeping character names and visual attributes stable across shots).
- Film Knowledge Graph (KG) Representation: The explicit bottleneck of the auto-encoder. It features 10 graph-addressable node types (Story, Event, Shot, Character, Scene, Object, Style, Camera, Audio, and Keyframe) and 11 explicit relation types. All nodes store structured text descriptions, while actual visual/audio assets are kept in a linked asset layer.
- Fixed Video Decoder: A static rendering pipeline that treats state-of-the-art text-to-image and image-to-video models as tools to reconstruct the video purely from the Film KG's text descriptions.
Dual-Loop Textual-Gradient Evolution
Because the video generation tools are non-differentiable, AVA-Encoder utilizes a dual-loop textual-gradient framework (inspired by TextGrad) to optimize the system using natural language feedback:
- The Outer Loop (Policy Pseudo-Training): Optimizes the general prompt instructions ($P_{\text{shot}}$ and $P_{\text{kf}}$) used by the encoder across a stream of training videos. It uses an Anti-Forgetting Gate to ensure updates don't degrade performance on previously learned cinematic rules.
- The Inner Loop (KG Representation Refinement): Operates at test time. It holds the prompt policy fixed and refines the input-specific text descriptions within the Film KG to correct localized reconstruction errors. An Anti-Degradation Gate ensures that fixing one detail doesn't inadvertently ruin another.
Datasets
The paper introduces two major resources:
- Film KG Dataset: The first dataset of its kind, containing text-based Film KG representations for tens of thousands of film shots derived from high-quality human films, award-winning shorts, and popular AI-generated videos.
- Agentic Video Representation Benchmark: A specialized evaluation split consisting of 18 non-overlapping video clips (comprising 129 shots and 246 keyframes). It evaluates reconstruction faithfulness across four distinct directions: direct Video comparison, direct Keyframe comparison, Video Back-Captioning, and Keyframe Back-Captioning.
Key Insights and Ideas
- Reconstruction as an Evaluation Metric: Instead of evaluating video understanding via standard downstream QA or retrieval tasks, the authors use video reconstruction fidelity. If a text-based graph contains enough information to faithfully recreate the original video, it proves the representation is truly cinematically faithful.
- Text as an Explicit Bottleneck: Forcing a high-dimensional video stream through a structured text graph creates an interpretable workspace. Because the graph explicitly maps production dependencies, agents can directly query, reason over, and modify the content.
- Gradients Without Math: By leveraging VLMs to answer atomic questions about the source vs. reconstructed video, the framework isolates exact factual discrepancies (e.g., "the character is wearing a blue shirt instead of a red one"). These text-based "residuals" serve as actionable gradients that guide an LLM to rewrite prompts or graph nodes.
Key Results
- State-of-the-Art Reconstruction: AVA-Encoder achieved an Overall score of 49.0% on the reconstruction benchmark, outperforming the strongest external baseline (soap2soap at 28.3%) by 20.7 percentage points absolute (a 73.1% relative improvement).
- Efficiency Gains: The pseudo-trained prompt policy outperformed a carefully human-tuned policy by 1.4 points while using 74.3% fewer shot-level tokens and 70.1% fewer keyframe-level tokens, proving that the automated outer loop learns highly concise and effective encoding rules.
- Topology-Aware Graph Editing: Because the Film KG tracks asset and production dependencies, a user or agent can perform local edits (like replacing a character identity or changing a visual style) that automatically propagate across all linked shots and keyframes while leaving unrelated subgraphs completely untouched.
Limitations and Future Work
- Inference Overhead: The test-time inner loop refinement requires multiple iterations of video generation and VLM evaluation, making it computationally expensive for real-time applications.
- Decoder Dependency: The final reconstruction quality is strictly bounded by the capabilities of the fixed generation tools. If the text-to-video tool hallucinates or fails to follow complex prompts, the auto-encoder's performance caps out, regardless of how perfect the Film KG representation is. The authors note that as video foundation models continue to mature, AVA-Encoder's reconstruction fidelity will scale naturally.
Summary updated