AVA-Encoder: Towards Agent-Native Video Representation Learning

Chuyue Li, Jinpeng Yu, Haozhe Wang, Tian Xueyun, Zhijing Zhang, Bingnan Li, Shuqi Gu, Kan Ren, Jiaming Liu, Ruihua HuangarXiv 2026 · arXiv

TL;DR

AVA-Encoder introduces an auto-encoding framework that bridges the gap between raw video and AI creative agents by translating films into a structured, text-based "Film Knowledge Graph" and back. Driven by an innovative dual-loop textual-gradient optimization process, it dramatically improves video reconstruction fidelity and enables precise, topology-aware video editing.

What Problem Does This Paper Address?

While foundation models have made massive strides in generating video content, AI video creation agents still struggle to produce truly cinematic-grade films. A core reason for this is a fundamental mismatch between video space and agent space.

Cinematic videos are dense, highly coupled multimodal streams where stories, character actions, camera movements, and audio are tightly intertwined across space and time. Conversely, AI agents learn and operate best within structured representations like text, code, and graphs. Existing video representations fail to bridge this gap: low-level visual tokens (pixels or latents) are uninterpretable for an agent to reason over, while standard textual captions or sparse scene graphs discard the fine-grained visual and production details required to actually recreate a scene.

Without a representation that is simultaneously agent-readable, editable, and cinematically faithful, agents cannot effectively study and learn filmmaking knowledge from high-quality human films.

Methods

To solve this, the authors propose the Agentic Video Auto-Encoder (AVA-Encoder). The framework operates like a classic auto-encoder but uses a structured text graph as its bottleneck and an LLM-driven optimization process instead of traditional backpropagation.

Overview of the AVA-Encoder framework, detailing the hierarchical encoder, the Film KG representation, and the dual-loop optimization process.

The system consists of three main components:

  1. Hierarchical Agentic Video Encoder: This module breaks down an input film into three granularities: film-level, shot-level, and keyframe-level. Crucially, it uses inter-level context injection, passing high-level narrative and entity registries down to finer stages to ensure global consistency (e.g., keeping character names and visual attributes stable across shots).
  2. Film Knowledge Graph (KG) Representation: The explicit bottleneck of the auto-encoder. It features 10 graph-addressable node types (Story, Event, Shot, Character, Scene, Object, Style, Camera, Audio, and Keyframe) and 11 explicit relation types. All nodes store structured text descriptions, while actual visual/audio assets are kept in a linked asset layer.
  3. Fixed Video Decoder: A static rendering pipeline that treats state-of-the-art text-to-image and image-to-video models as tools to reconstruct the video purely from the Film KG's text descriptions.

Dual-Loop Textual-Gradient Evolution

Because the video generation tools are non-differentiable, AVA-Encoder utilizes a dual-loop textual-gradient framework (inspired by TextGrad) to optimize the system using natural language feedback:

  • The Outer Loop (Policy Pseudo-Training): Optimizes the general prompt instructions ($P_{\text{shot}}$ and $P_{\text{kf}}$) used by the encoder across a stream of training videos. It uses an Anti-Forgetting Gate to ensure updates don't degrade performance on previously learned cinematic rules.
  • The Inner Loop (KG Representation Refinement): Operates at test time. It holds the prompt policy fixed and refines the input-specific text descriptions within the Film KG to correct localized reconstruction errors. An Anti-Degradation Gate ensures that fixing one detail doesn't inadvertently ruin another.

Datasets

The paper introduces two major resources:

  • Film KG Dataset: The first dataset of its kind, containing text-based Film KG representations for tens of thousands of film shots derived from high-quality human films, award-winning shorts, and popular AI-generated videos.
  • Agentic Video Representation Benchmark: A specialized evaluation split consisting of 18 non-overlapping video clips (comprising 129 shots and 246 keyframes). It evaluates reconstruction faithfulness across four distinct directions: direct Video comparison, direct Keyframe comparison, Video Back-Captioning, and Keyframe Back-Captioning.

Key Insights and Ideas

  • Reconstruction as an Evaluation Metric: Instead of evaluating video understanding via standard downstream QA or retrieval tasks, the authors use video reconstruction fidelity. If a text-based graph contains enough information to faithfully recreate the original video, it proves the representation is truly cinematically faithful.
  • Text as an Explicit Bottleneck: Forcing a high-dimensional video stream through a structured text graph creates an interpretable workspace. Because the graph explicitly maps production dependencies, agents can directly query, reason over, and modify the content.
  • Gradients Without Math: By leveraging VLMs to answer atomic questions about the source vs. reconstructed video, the framework isolates exact factual discrepancies (e.g., "the character is wearing a blue shirt instead of a red one"). These text-based "residuals" serve as actionable gradients that guide an LLM to rewrite prompts or graph nodes.

Key Results

  • State-of-the-Art Reconstruction: AVA-Encoder achieved an Overall score of 49.0% on the reconstruction benchmark, outperforming the strongest external baseline (soap2soap at 28.3%) by 20.7 percentage points absolute (a 73.1% relative improvement).
  • Efficiency Gains: The pseudo-trained prompt policy outperformed a carefully human-tuned policy by 1.4 points while using 74.3% fewer shot-level tokens and 70.1% fewer keyframe-level tokens, proving that the automated outer loop learns highly concise and effective encoding rules.
  • Topology-Aware Graph Editing: Because the Film KG tracks asset and production dependencies, a user or agent can perform local edits (like replacing a character identity or changing a visual style) that automatically propagate across all linked shots and keyframes while leaving unrelated subgraphs completely untouched.
Visual example of graph-topology identity editing across a sequence of shots.

Limitations and Future Work

  • Inference Overhead: The test-time inner loop refinement requires multiple iterations of video generation and VLM evaluation, making it computationally expensive for real-time applications.
  • Decoder Dependency: The final reconstruction quality is strictly bounded by the capabilities of the fixed generation tools. If the text-to-video tool hallucinates or fails to follow complex prompts, the auto-encoder's performance caps out, regardless of how perfect the Film KG representation is. The authors note that as video foundation models continue to mature, AVA-Encoder's reconstruction fidelity will scale naturally.

Summary updated

Abstract

Video creative agents still lack an effective way to learn from high-quality human films, limiting their ability to produce cinematic-grade videos. A key challenge is the absence of a structured video representation that is both faithful to film content and directly usable for agentic reasoning and manipulation. To address the challenge, we propose the Agentic Video Auto-Encoder (AVA-Encoder), a novel auto-encoding framework driven by agentic self-evolution to learn agent-native video representations. AVA-Encoder transforms a video into a Film Knowledge Graph (KG) representation and then reconstructs it back into video. This Film KG representation explicitly captures entities, events, assets, and their multimodal relationships in a structured form that can be easily understood, queried, and manipulated by agents. The reconstruction residual drives a dual-loop textual-gradient optimization framework that jointly improves the Film KG representation and the Agentic Video Encoder. Extensive experiments show that AVA-Encoder achieves a 20.7-percentage-point absolute gain, or a 73.1% relative improvement, over the strongest external baseline. In the controlled policy-only setting, its pseudo-trained Agentic Video Encoder policy also outperforms a carefully human-tuned policy while using 74.3% fewer shot-level and 70.1% fewer keyframe-level system-prompt tokens. We release the complete AVA-Encoder framework, a reliable agentic video reconstruction benchmark, and the first dataset of high-quality Film KG representations.
BibTeX
@misc{li2026ava,
  author = {Chuyue Li and Jinpeng Yu and Haozhe Wang and Tian Xueyun and Zhijing Zhang and Bingnan Li and Shuqi Gu and Kan Ren and Jiaming Liu and Ruihua Huang},
  title = {{AVA-Encoder: Towards Agent-Native Video Representation Learning}},
  year = {2026},
  eprint = {2608.12313},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2608.12313},
}
Figure