Real-time motion control using pose space probability density estimation

Dumebi Okwechime, Eng-Jon Ong, Richard BowdenWorkshops @ ICCV 2009 · OpenReview

Abstract

The ability to control the movements of an object or person in a video sequence has significant applications in the movie industry, animation, and Human-Computer Interaction (HCI). This paper introduces a novel algorithm for real-time motion control by modeling video frames as a probability density function (PDF) within a low-dimensional "pose space." By combining unsupervised clustering with fast density estimation, the system allows users to intuitively drive video synthesis—such as controlling a candle flame or generating realistic facial reactions to speech—while maintaining the natural style and variance of the original footage. We demonstrate the flexibility of the model across datasets composed of both vectorized images and 2D point representations, showing plausible motion generation and lifelike blends between different types of movement.

Introduction

Creating new video sequences from pre-recorded footage (video synthesis) is a major challenge in animation and HCI. While techniques like "Video Textures" exist, they often struggle with real-time control and computational efficiency. Standard methods often require manual segmentation of video clips or exhaustive frame-by-frame distance calculations, which do not scale well to large datasets. Furthermore, making these synthesized videos interactive—where a user or an external signal (like audio) can steer the motion in real-time—is difficult to achieve without the motion looking robotic or repetitive. This research seeks a way to generate "plausible" novel motion that feels as authentic as the original source material but remains fully controllable.

Methods

The proposed Motion Control Model consists of five primary stages:

  1. Dimensionality Reduction: To handle high-dimensional video data (pixels or 2D points), the system uses Principal Component Analysis (PCA) to project frames into a lower-dimensional "pose space," retaining 95% of the data's energy while accelerating calculations.
  2. Unsupervised Segmentation: Instead of manual labeling, a k-medoids clustering algorithm identifies "cut points." These are regions in the pose space where different motion sequences intersect, representing logical points where the video can transition.
  3. Pose Space PDF: The system models transition likelihoods using a Gaussian Mixture Model (GMM). To maintain real-time performance, kd-trees are employed for fast Gaussian approximation.
  4. Markov Transitions & Dynamic Programming: A first-order Markov Transition Matrix ensures generated paths are physically inherent to the training data. Dynamic Programming is used to look three steps ahead, ensuring smooth, long-term motion.
  5. Projection Mapping: This control layer maps external inputs (like a mouse cursor or audio features) to conditional probabilities in the pose space, effectively weighting the PDF to favor desired motions.
Facial video frames and their corresponding 2D point representations used for tracking.

Datasets and Experimental Setup

The researchers evaluated their model on three distinct types of data to demonstrate its flexibility:

  • Candle Flame: 3,000 frames (15 fps) of a flame being blown left, right, or burning steadily.
  • Plasma Beam: 2,985 frames of a plasma ball with varying degrees of concentrated beam movement.
  • Conversational Data: A 12-minute recording of two people talking. The authors extracted 2:30 minutes of "active listening" footage, using 44 2D tracked points for the face and Mel-Frequency Cepstral Coefficients (MFCCs) to represent the speaker's audio.
Sample frames from the candle flame and plasma ball datasets used to test qualitative control.

Key Insights and Results

Motion as a Probability Problem

Rather than seeing video synthesis as "stitching clips," the authors treat it as navigating a continuous probability landscape. This allows for lifelike blends rather than jarring jumps between frames. The "cut point" strategy identifies hidden transition points that a human editor might miss, increasing the variety of possible novel sequences.

Real-Time Performance and Control

For the candle and plasma datasets, the system successfully allowed users to steer the motion (e.g., making the flame blow left on command) while ensuring the flame always passed through a stationary state to maintain physical realism. The shift from exhaustive search to kd-tree-based density estimation is what transforms this from an offline rendering tool into a real-time interactive system.

Audio-Driven Synthesis: The "Synthetic Listener"

In the conversational test, the model generated head nods and facial expressions in response to novel audio. By learning the relationship between audio features (MFCCs) and facial poses, the system can generate appropriate non-verbal cues. In user validation, 14 participants rated the model against a random playback baseline. The model achieved a normalized score of 0.62, significantly outperforming the baseline (0.37), with 11 out of 14 participants preferring the model-generated responses.

Limitations and Future Work

The success of the model is heavily dependent on the size and variety of the training database; small datasets can lead to repetitive motion. While the system handles head nods and blinks well, future work should extend to more complex non-verbal communication, such as hand gestures. Additionally, while the model outperformed random playback, further refinement of audio-visual mapping is needed to achieve true human-level conversational realism.

Summary updated

Abstract

The ability to control the movements of an object or person in a video sequence has applications in the movie and animation industries, and in HCI. In this paper, we introduce a new algorithm for real-time motion control and demonstrate its application to pre-recorded video clips and HCI. Firstly, a dataset of video frames are projected into a lower dimension space. A k-mediod clustering algorithm with a distance metric is used to determine groups of similar frames which operate as cut points, segmenting the data into smaller subsequences. A multivariate probability distribution is learnt and probability density estimation is used to determine transitions between the subsequences to develop novel motion. To facilitate real-time control, conditional probabilities are used to derive motion given user commands. The motion controller is extended to HCI using speech Mel-Frequency Ceptral Coefficients (MFCCs) to trigger movement from an input speech signal. We demonstrate the flexibility of the model by presenting results ranging from datasets composed of both vectorised images and 2D point representation. Results show plausible motion generation and lifelike blends between different types of movement.
BibTeX
@inproceedings{okwechime2009real,
  author = {Dumebi Okwechime and Eng-Jon Ong and Richard Bowden},
  title = {{Real-time motion control using pose space probability density estimation}},
  booktitle = {Workshops @ ICCV},
  year = {2009},
  url = {https://openreview.net/forum?id=lVDt1fTSTS},
}
Figure