Smooth and Flexible Camera Movement Synthesis via Temporal Masked Generative Modeling

Chenghao Xu, Guangtao Lyu, Jiexi Yan, Muli Yang, Cheng DengNeurIPS 2025 · OpenReview

Overview

TemMEGA introduces a novel framework for online camera movement synthesis, allowing AI to generate cinematic camera trajectories in real-time during live dance performances. By combining a discrete camera tokenizer with a hierarchical memory encoder, the system produces smooth, music-synced camera work using only past and current movement data. This approach significantly outperforms existing "offline" models that require the entire performance in advance.

The Challenge: Offline vs. Online Synthesis

In the world of dance cinematography, the camera is not just a passive observer; it is a storyteller that must synchronize with the rhythm of the music and the flow of the choreography. While recent AI models have made strides in automating this process, they suffer from a major "look-ahead" bias. These models are predominantly offline, meaning they need the entire dance sequence from start to finish before they can generate a corresponding camera path.

Traditional offline camera movement generation requires the entire dance and music sequence in advance, limiting real-time application.

This constraint makes them impractical for live stage performances or interactive applications where the dance is happening in the moment. To address this, the authors propose a shift to online camera movement synthesis, where the system must predict the next cinematic move based only on the history of the performance so far. The primary challenge is maintaining temporal smoothness and artistic vision without knowing what the dancer will do seconds into the future.

The TemMEGA Framework

The authors present TemMEGA (Temporal Masked Generative Modeling), which breaks the complex problem of real-time camera synthesis into three sophisticated stages:

  1. Discrete Camera Tokenizer (DCT): Instead of treating camera coordinates as raw continuous numbers, TemMEGA uses a VQ-VAE (Vector Quantized Variational Autoencoder) to "tokenize" camera movements. It maps complex 3D camera paths—including position, rotation, and field of view—into a discrete vocabulary of motion tokens. This allows the model to treat camera synthesis similarly to how a Large Language Model (LLM) processes text.
  2. Consecutive Memory Encoder (CME): To handle the "online" constraint, the model needs a robust sense of history. The CME splits past data into short-term memory (recent frames for immediate reaction) and long-term memory (older frames for overall style). Crucially, it compresses the long-term history into abstract features to keep computation efficient while retaining the overarching "vibe" of the performance.
  3. Temporal Conditional Masked Transformer (CMT): This serves as the generative engine. It uses a bidirectional transformer to predict "masked" (future) camera tokens, conditioned on the dance motion, music features, and the hierarchical memory from the CME. To ensure the camera stays focused on the dancer, it uses Adaptive Instance Normalization (AdaIN) to inject long-term style and cross-attention to align with short-term movements.

Data and Training

The study primarily utilizes the DCM (3D Dance-Camera-Music) dataset, which contains 108 pieces of professionally designed dance-camera-music pairings. The data includes:

  • Dance Data: Global positions of 60 human joints.
  • Music Data: 35-dimensional features extracted via the Librosa library.
  • Camera Data: MMD-format polar coordinates (reference point, rotation, distance, and FOV).

The authors adapted this dataset for the online task by stitching adjacent data pieces together, providing longer historical contexts for training the memory modules.

Core Innovations

  • The Power of Tokenization: By converting 3D movement into discrete tokens, the model avoids the "drifting" issues often seen in continuous regression models. It forces the AI to choose from a learned vocabulary of valid cinematic moves.
  • Memory Hierarchy: The architecture demonstrates that high-resolution data from the distant past isn't necessary for immediate decisions. Compressing long-term history into macro-level features (like overall energy) while keeping short-term history at a micro-level (exact limb positions) provides an optimal balance of stability and responsiveness.
  • Causal Masking for Real-Time: During inference, the model uses a specific [MASK] token at the current time step. This ensures the model never "cheats" by looking at future frames, proving that high-quality cinematography is possible with zero look-ahead.

Performance and Results

TemMEGA sets a new state-of-the-art (SOTA) for both online and offline settings:

  • Quality & Diversity: In online tests, TemMEGA achieved a Kinetic FID ($FID_k$) of 4.025, significantly outperforming the previous SOTA, DanceCamera3D (4.634).
  • Dancer Fidelity: It excelled at keeping the dancer in frame, achieving the lowest Dancer Missing Rate (DMR) and Limbs Capture Difference (LCD). This indicates the AI "cameraman" is highly effective at tracking the performer.
  • User Preference: In a blind study, human participants—including filmmakers and dancers—preferred TemMEGA’s camera work over baselines more than 83% of the time in online scenarios.

Future Directions

While TemMEGA is highly flexible, the authors acknowledge it is currently trained on a relatively small dataset. Expanding the diversity of dance styles and camera "languages" is a clear next step. Additionally, the model currently focuses on a single-camera setup. Future work could involve multi-camera coordination, where the system decides when to "cut" between different virtual cameras to maximize the dramatic impact of a live performance, or navigating complex 3D environments with physical obstacles.

Summary updated

Abstract

In dance performances, choreographers define the visual expression of movement, while cinematographers shape its final presentation through camera work. Consequently, the synthesis of camera movements informed by both music and dance has garnered increasing research interest. While recent advancements have led to notable progress in this area, existing methods predominantly operate in an offline manner—that is, they require access to the entire dance sequence before generating corresponding camera motions. This constraint renders them impractical for real-time applications, particularly in live stage performances, where immediate responsiveness is essential. To address this limitation, we introduce a more practical yet challenging task: online camera movement synthesis, in which camera trajectories must be generated using only the current and preceding segments of dance and music. In this paper, we propose TemMEGA (Temporal Masked Generative Modeling), a unified framework capable of handling both online and offline camera movement generation. TemMEGA consists of three key components. First, a discrete camera tokenizer encodes camera motions as discrete tokens via a discrete quantization scheme. Second, a consecutive memory encoder captures historical context by jointly modeling long- and short-term temporal dependencies across dance and music sequences. Finally, a temporal conditional masked transformer is employed to predict future camera motions by leveraging masked token prediction. Extensive experimental evaluations demonstrate the effectiveness of our TemMEGA, highlighting its superiority in both online and offline camera movement synthesis.
BibTeX
@inproceedings{xu2025smooth,
  author = {Chenghao Xu and Guangtao Lyu and Jiexi Yan and Muli Yang and Cheng Deng},
  title = {{Smooth and Flexible Camera Movement Synthesis via Temporal Masked Generative Modeling}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2025},
  url = {https://openreview.net/forum?id=LujP8dao7o},
}
Figure