Abstract
Click to Move (C2M) is a framework designed to provide intuitive control over video generation from static images. By utilizing sparse user inputs—simple mouse clicks and drags—C2M generates realistic, temporally consistent videos. The core of the system is a Graph Convolutional Network (GCN) that models object interactions, ensuring that the entire scene reacts plausibly to the movement of specific entities.
Introduction
Generating video from a single image remains a significant challenge in computer vision, particularly when trying to provide users with fine-grained control. While existing models can animate simple subjects, complex urban environments with multiple interacting objects often suffer from "ghosting" or static artifacts where objects remain frozen despite user input.
The C2M framework addresses this by allowing sparse user input to drive dense video synthesis. Instead of requiring complex instructions, a user simply clicks an object and drags it to a destination. The system then predicts the motion of all objects in the scene, maintaining structural integrity and realistic interactions.
Methodology
The C2M architecture is composed of three primary components: Appearance Encoding, Motion Encoding, and a Generation Module.
1. Appearance Encoding
This module processes the initial frame alongside its segmentation and instance maps. It extracts high-level features that define the visual characteristics of the scene, ensuring the generated video maintains the "look and feel" of the original image.
2. Motion Encoding via GCN
The central innovation of C2M is its use of a Graph Convolutional Network (GCN) to model object interactions. Each object in the scene is represented as a node. When a user provides a motion vector for one object, the GCN propagates this information throughout the graph. This allows the model to infer how other objects should move or adjust, even without direct user input. To handle the inherent uncertainty of motion, this process is integrated into a Variational Auto-Encoder (VAE).
3. Generation Module
The final stage uses the predicted optical flow and appearance features to warp the initial frame. It synthesizes a sequence of frames that are both temporally consistent and visually realistic.
Datasets and Evaluation
The framework was evaluated on the Cityscapes and KITTI 360 datasets, which provide diverse urban and suburban street scenes. Since perfect instance segmentations are not always available, the authors utilized state-of-the-art pre-trained models to generate the necessary training data.
Key Insights and Results
The research highlights that object motion is inherently social; objects do not move in isolation. The GCN-based approach successfully transforms sparse 2D vectors into dense, pixel-level optical flow.
Quantitative Performance
- Video Quality: C2M achieved a 22.8% improvement in Fréchet Video Distance (FVD) on the Cityscapes dataset compared to previous state-of-the-art methods.
- Motion Accuracy: Using Normalized Displacement Error (NDE), the authors demonstrated that C2M accurately follows user-defined paths, whereas baseline models often failed to move objects or introduced significant distortions.
Limitations and Future Work
Currently, C2M generates fixed-length sequences of five frames and is sensitive to the quality of initial segmentation maps. Future research will focus on extending video length, handling complex occlusions, and improving robustness to segmentation errors.
The source code for C2M is available on GitHub.
Summary updated