Click to Move: Controlling Video Generation with Sparse Motion

Pierfrancesco Ardino, Marco De Nadai, Bruno Lepri, Elisa Ricci, Stéphane Lathuilière

Abstract

Click to Move (C2M) is a framework designed to provide intuitive control over video generation from static images. By utilizing sparse user inputs—simple mouse clicks and drags—C2M generates realistic, temporally consistent videos. The core of the system is a Graph Convolutional Network (GCN) that models object interactions, ensuring that the entire scene reacts plausibly to the movement of specific entities.

Introduction

Generating video from a single image remains a significant challenge in computer vision, particularly when trying to provide users with fine-grained control. While existing models can animate simple subjects, complex urban environments with multiple interacting objects often suffer from "ghosting" or static artifacts where objects remain frozen despite user input.

The Click to Move (C2M) workflow: from user input to generated video.

The C2M framework addresses this by allowing sparse user input to drive dense video synthesis. Instead of requiring complex instructions, a user simply clicks an object and drags it to a destination. The system then predicts the motion of all objects in the scene, maintaining structural integrity and realistic interactions.

Methodology

The C2M architecture is composed of three primary components: Appearance Encoding, Motion Encoding, and a Generation Module.

The C2M architecture, featuring Appearance and Motion Encoding modules and a Graph Convolutional Network (GCN).

1. Appearance Encoding

This module processes the initial frame alongside its segmentation and instance maps. It extracts high-level features that define the visual characteristics of the scene, ensuring the generated video maintains the "look and feel" of the original image.

2. Motion Encoding via GCN

The central innovation of C2M is its use of a Graph Convolutional Network (GCN) to model object interactions. Each object in the scene is represented as a node. When a user provides a motion vector for one object, the GCN propagates this information throughout the graph. This allows the model to infer how other objects should move or adjust, even without direct user input. To handle the inherent uncertainty of motion, this process is integrated into a Variational Auto-Encoder (VAE).

3. Generation Module

The final stage uses the predicted optical flow and appearance features to warp the initial frame. It synthesizes a sequence of frames that are both temporally consistent and visually realistic.

Datasets and Evaluation

The framework was evaluated on the Cityscapes and KITTI 360 datasets, which provide diverse urban and suburban street scenes. Since perfect instance segmentations are not always available, the authors utilized state-of-the-art pre-trained models to generate the necessary training data.

Key Insights and Results

The research highlights that object motion is inherently social; objects do not move in isolation. The GCN-based approach successfully transforms sparse 2D vectors into dense, pixel-level optical flow.

Quantitative Performance

  • Video Quality: C2M achieved a 22.8% improvement in Fréchet Video Distance (FVD) on the Cityscapes dataset compared to previous state-of-the-art methods.
  • Motion Accuracy: Using Normalized Displacement Error (NDE), the authors demonstrated that C2M accurately follows user-defined paths, whereas baseline models often failed to move objects or introduced significant distortions.

Limitations and Future Work

Currently, C2M generates fixed-length sequences of five frames and is sensitive to the quality of initial segmentation maps. Future research will focus on extending video length, handling complex occlusions, and improving robustness to segmentation errors.

The source code for C2M is available on GitHub.

Summary updated

Abstract

This paper introduces Click to Move (C2M), a novel framework for video generation where the user can control the motion of the synthesized video through mouse clicks specifying simple object trajectories of the key objects in the scene. Our model receives as input an initial frame, its corresponding segmentation map and the sparse motion vectors encoding the input provided by the user. It outputs a plausible video sequence starting from the given frame and with a motion that is consistent with user input. Notably, our proposed deep architecture incorporates a Graph Convolution Network (GCN) modelling the movements of all the objects in the scene in a holistic manner and effectively combining the sparse user motion information and image features. Experimental results show that C2M outperforms existing methods on two publicly available datasets, thus demonstrating the effectiveness of our GCN framework at modelling object interactions. The source code is publicly available at https://github.com/PierfrancescoArdino/C2M.
BibTeX
@inproceedings{ardino2021click,
  author = {Pierfrancesco Ardino and Marco De Nadai and Bruno Lepri and Elisa Ricci and Stéphane Lathuilière},
  title = {{Click to Move: Controlling Video Generation with Sparse Motion}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2021},
  doi = {10.1109/iccv48922.2021.01448},
}
Figure