Compositional Video Synthesis with Action Graphs

Amir Bar, Roei Herzig, Xiaolong Wang, Gal Chechik, Trevor Darrell, Amir Globerson

TL;DR

This paper introduces "Action Graph to Video" (AG2Vid), a novel approach for generating videos that accurately depict coordinated and timed actions between multiple objects. By representing actions as a structured "Action Graph," AG2Vid enables fine-grained control over video synthesis, outperforming existing methods in both visual quality and semantic accuracy.

What Problem Does This Paper Address?

Generating realistic videos, especially those involving complex interactions between multiple objects and agents, remains a significant challenge. Existing video generation methods often struggle to handle the intricate spatio-temporal dependencies and simultaneous actions that characterize real-world scenarios. While some approaches can generate videos based on single actions or semantic masks, they lack the ability to precisely control the timing and coordination of multiple, potentially overlapping, actions. This limitation hinders applications requiring nuanced control, such as training autonomous agents or creating realistic simulations. This paper aims to bridge this gap by proposing a structured representation for actions and a generative model capable of synthesizing videos that adhere to these complex action specifications.

Methods

The core innovation of this work is the Action Graph (AG), a graph-based representation where nodes represent objects and directed edges represent actions. Each action edge is annotated with its start and end times, allowing for precise temporal control. The proposed model, AG2Vid, leverages this AG to synthesize videos.

AG2Vid operates in three stages:

  1. Action Scheduling with "Clocked Edges": To manage the temporal progression of actions, each action edge in the AG is augmented with a "clock" that tracks its progress from 0 to 1. This creates a sequence of AGs, one for each time step, facilitating timely and coordinated action execution.
  2. Layout Generation Function (LGF): A Graph Convolutional Network (GCN) is employed to predict a sequence of intermediate "layouts" (object positions and descriptors) based on the current AG and the previous layout. The GCN effectively propagates information across objects and actions, capturing their interactions.
  3. Frame Generation Function (FGF): The predicted layout, along with the previous frame and layout, is used to generate the next frame. This involves transforming layouts into feature maps, estimating optical flow for motion, and then using a network (SPADE generator) to refine the warped frame.

The model is trained using a combination of losses: a layout prediction loss (L1), a pixel action discriminator loss (GAN-based), and a flow loss.

Datasets

The authors evaluate AG2Vid on two datasets:

  • CATER: A synthetic video dataset designed for action recognition and reasoning. It provides object annotations, attributes, action labels, and timings. The authors use this dataset to create Action Graphs for training and evaluation, including scenarios with multiple simultaneous actions.
  • Something-Something V2 (SmthV2): A dataset of real-world videos of humans performing basic actions. The authors use the eight most frequent actions and object bounding box annotations.

Key Insights and Ideas

  • Action Graphs as a Structured Representation: The AG provides a powerful and intuitive way to represent complex, timed, and coordinated actions, overcoming the limitations of simpler action representations.
  • "Clocked Edges" for Temporal Control: The introduction of "clocked edges" is a clever mechanism for explicitly modeling and controlling the temporal progression of actions, crucial for realistic video synthesis.
  • GCN for Layout Prediction: Using a GCN to process the Action Graph allows the model to effectively capture inter-object and inter-action dependencies, leading to more accurate layout predictions.
  • Zero-Shot Compositionality: The AG2Vid model demonstrates impressive zero-shot generalization capabilities, allowing it to synthesize novel compositions of learned actions that were not seen during training. This highlights the compositional power of the AG representation.

Key Results

  • Human Evaluation: AG2Vid significantly outperforms baseline methods (CVP, HG, V2V, RNN) in both semantic accuracy and visual quality, as judged by human annotators. For example, in Table 1, AG2Vid is selected as better in a high percentage of comparisons.
  • Quantitative Metrics: Table 3 shows AG2Vid achieving competitive or superior results on visual quality metrics like Inception Score and FID (where applicable) compared to baselines.
  • Layout Accuracy: The GCN-based LGF in AG2Vid achieves a mean Intersection Over Union (mIOU) of 51.3% on CATER and 74.5% on SmthV2 for layout generation, significantly outperforming an RNN baseline.
  • Action Timing: Human evaluation shows AG2Vid correctly executes actions at the intended times in 89.45% of cases.
  • Zero-Shot Action Composition: The model achieves high class recall (e.g., 96.65% on CATER, 87.5% on SmthV2) for novel, composed actions.

Limitations and Future Work

  • Occlusion and Overlapping Objects: AG2Vid still struggles in cases with occluded or overlapping objects. The authors suggest that incorporating more fine-grained intermediate representations like segmentation masks could help.
  • Rigid vs. Non-Rigid Motion: The current model excels at synthesizing rigid motion. For non-rigid motion, conditioning on random noise or explicitly defining action attributes like speed or style is suggested for future work.
  • Scene Graph Comparison: While AG2Vid outperforms scene-graph-based generation in terms of temporal coherence, the latter is noted as being static and not modeling object motion.
  • Action Graph Interpretation: The authors provide evidence that AG2Vid indeed uses the Action Graph by demonstrating that changing the AG while keeping the first frame and layout fixed results in different video outputs.

Summary updated

Abstract

Videos of actions are complex signals containing rich compositional structure in space and time. Current video generation methods lack the ability to condition the generation on multiple coordinated and potentially simultaneous timed actions. To address this challenge, we propose to represent the actions in a graph structure called Action Graph and present the new ``Action Graph To Video'' synthesis task. Our generative model for this task (AG2Vid) disentangles motion and appearance features, and by incorporating a scheduling mechanism for actions facilitates a timely and coordinated video generation. We train and evaluate AG2Vid on the CATER and Something-Something V2 datasets, and show that the resulting videos have better visual quality and semantic consistency compared to baselines. Finally, our model demonstrates zero-shot abilities by synthesizing novel compositions of the learned actions. For code and pretrained models, see the project page https://roeiherz.github.io/AG2Video
BibTeX
@inproceedings{bar2021compositional,
  author = {Amir Bar and Roei Herzig and Xiaolong Wang and Gal Chechik and Trevor Darrell and Amir Globerson},
  title = {{Compositional Video Synthesis with Action Graphs}},
  booktitle = {International Conference on Machine Learning},
  year = {2021},
  url = {https://openreview.net/forum?id=gdM8o4QjLZ},
}
Figure