Introduction
Generating realistic videos is incredibly difficult, but conditional video generation—where a user dictates exactly what happens in the scene—is even harder. Current methods typically condition video generation on a single action class, a text prompt, or frame-by-frame semantic segmentation maps.
However, these approaches fall short when trying to generate complex scenes with multiple agents performing coordinated, simultaneous, and precisely timed actions (e.g., a basketball play where one player passes while another sets a screen). Text prompts can be ambiguous regarding exact timing, and semantic maps lack explicit action information.
To tackle this gap, researchers introduce a novel task and generative model (AG2Vid) for synthesizing videos based on "Action Graphs"—a highly structured, object-centric representation of multiple, coordinated, and timed actions.
By explicitly controlling who is doing what, and exactly when they are doing it, the model generates complex multi-agent videos that significantly outperform existing baselines in both visual quality and semantic accuracy, even demonstrating zero-shot generalization to novel action combinations.
Methodology: The AG2Vid Model
The core of this paper is the "Action Graph to Video" task. The input consists of an initial video frame, its object layout, and an Action Graph (AG). In an AG, nodes represent objects and directed edges represent actions, annotated with specific start and end times.
To turn this graph into a realistic video sequence, the AG2Vid model utilizes three levels of abstraction:
- Clocked Edges: To handle the passage of time, the model augments the Action Graph with a scheduling mechanism. Each action edge gets a continuous "clock" value between 0 and 1, representing the exact progress of that action at any given frame.
- Layout Generating Function (LGF): Instead of jumping straight to pixels, a Graph Convolutional Network (GCN) processes the clocked Action Graph and the previous frame's layout to predict the next frame's bounding box layout. This crucial step disentangles motion planning from pixel rendering.
- Frame Generating Function (FGF): The predicted layouts are converted into spatial feature maps. The model then predicts optical flow between the frames, warps the previous frame's pixels to their new locations, and refines the output using a SPADE generator to produce the final image.
Datasets and Evaluation
The model was trained and evaluated on two datasets:
- CATER: A synthetic video dataset featuring multiple objects (spheres, cones, etc.) performing simultaneous actions like "Rotate," "Contain," "Pick Place," and "Slide." This is ideal for testing complex, multi-action coordination.
- Something-Something V2 (SmthV2): A dataset of real-world videos showing humans performing basic actions (e.g., "Push Left," "Move Down"). While the training videos here only contain single actions, the authors use it to test the model's ability to generate zero-shot multi-action compositions at inference time.
Key Insights and Results
- Explicit Timing is Crucial: By explicitly modeling time as a continuous "clock" on graph edges, the model can smoothly interpolate and coordinate actions that unfold over multiple frames. It follows a strict schedule rather than just guessing the next frame.
- Intermediate Layouts Bridge the Semantic Gap: Directly generating pixels from a high-level graph is too difficult. Predicting intermediate bounding box layouts acts as a crucial stepping stone, separating where things should move from what they look like.
- Compositional Generalization: Because the model learns atomic actions via a structured graph, it can successfully synthesize zero-shot "composite" actions at test time. For example, it can combine a learned "Pick-Place" action and a "Slide" action to generate a completely novel "Swap" action that it never saw during training. When tasked with generating entirely new composite actions, the model achieved an average class recall of 96.65% on CATER and 87.5% on SmthV2.
- Superior Visual and Semantic Quality: In head-to-head human trials for semantic accuracy, AG2Vid crushed the baselines. On the CATER dataset, human raters preferred AG2Vid over the CVP baseline 85.7% of the time, and 90.6% of the time on SmthV2. AG2Vid also achieved significantly better LPIPS scores (a metric for perceptual image quality) than baselines: 0.04 vs 0.24 (CVP) on CATER.
- Precise Timing Control: When researchers generated pairs of videos where the only difference was the scheduled start time of an action, human annotators confirmed the model correctly followed the timing instructions in 89.45% of the cases.
Limitations and Future Work
The authors acknowledge a few limitations. First, AG2Vid struggles when synthesizing occluded or overlapping objects, as bounding boxes don't provide enough detail to resolve depth and boundaries perfectly. Second, the model is currently best suited for rigid motion; synthesizing actions with complex, non-rigid motion (like detailed human body articulation) remains difficult.
For future work, the authors suggest incorporating more fine-grained intermediate representations, such as semantic segmentation masks, to alleviate occlusion issues. They also propose conditioning the model on random noise or explicit style attributes to better capture non-rigid or stochastic motions.
Summary updated