Beyond Skeletons: Integrative Latent Mapping for Coherent 4D Sequence Generation

Qitong Yang, Mingtao Feng, Zijie Wu, Shijie Sun, Weisheng Dong, Yaonan Wang, Ajmal MianarXiv 2024 · arXiv · OpenReview

Abstract

Directly learning to model 4D content, including shape, color, and motion, is challenging. Existing methods depend on skeleton-based motion control and offer limited continuity in detail. To address this, we propose a novel framework that generates coherent 4D sequences with animation of 3D shapes under given conditions with dynamic evolution of shape and color over time through integrative latent mapping. We first employ an integrative latent unified representation to encode shape and color information of each detailed 3D geometry frame. The proposed skeleton-free latent 4D sequence joint representation allows us to leverage diffusion models in a low-dimensional space to control the generation of 4D sequences. Finally, temporally coherent 4D sequences are generated conforming well to the input images and text prompts.

Introduction

Generating 4D content—3D geometry that evolves over time—is a massive challenge in computer vision. Current approaches usually fall into two camps: prior-based methods that rely on restrictive skeletal rigs (like SMPL), and free-form methods (like NeRF or Gaussian Splatting) that often suffer from temporal flickering or a lack of underlying geometric structure. This paper identifies a "representation gap" and proposes a solution that enables efficient spatio-temporal diffusion to produce coherent animations without relying on pre-defined skeletons.

Figure 1: A sequence of generated frames demonstrating the model's ability to maintain character identity and geometric coherence over time.

Methodology

The proposed framework bridges the representation gap through a three-stage approach:

1. Coherent 3D Shape and Color Modeling

Instead of treating a 3D frame as a single blob of data, the system decouples it into a Signed Distance Function (SDF) for shape and a Continuous Color Field for appearance. A 3D U-Net refines the coarse SDF to capture fine details. This decoupling allows for independent editing, such as changing a character's clothes without altering its shape.

2. Matrixized 4D Representation

To handle the temporal dimension without exploding computational costs, the authors concatenate the latent vectors of keyframes into a matrix $\mathcal{M}$. Intermediate frames are recovered via latent space interpolation. This "matrixized" approach captures the 4D distribution of a sequence more efficiently than processing every frame individually, reducing complexity from $O(T^2)$ to $O(\frac{T^2}{K^2})$.

3. Hierarchical Conditional Spatio-Temporal Attention (HCSTA)

A specialized diffusion block uses a staged approach to ensure consistency:

  • Shape-Color Cross-Attention: Ensures the color "sticks" to the geometry.
  • Conditioned Cross-Attention: Injects the text and image prompts.
  • Temporal Self-Attention: Enforces consistency across the timeline.
Figure 2: Overview of the proposed 4D generation framework, showing conditional inputs (image and text) and the resulting volumetric sequence across multiple views and time frames.

Experimental Results

The model was evaluated on diverse benchmarks including ShapeNet, 3DBiCar, and DeformingThings4D (DT4D). The results demonstrate significant improvements in both speed and quality:

  • Generation Speed: The model generates a 4D sequence in approximately 0.625 seconds, significantly faster than DreamGaussian4D (~6.5 minutes) or STAG4D (~7.5 minutes).
  • Quantitative Superiority: On the 3DBiCar and DT4D datasets, the method achieved the best (lowest) Minimum Matching Distance (MMD) and highest Coverage (COV), indicating more realistic and diverse generations.
  • Qualitative Fidelity: User studies rated the model higher than competitors in terms of spatio-temporal consistency and alignment with input prompts, producing smoother surfaces and more detailed features.

Key Insights and Discussion

A critical advantage of this method is Geometry-Aware Color. By linking the color field to SDF values, the model ensures color remains consistent even as the object deforms, preventing the "color drifting" common in other methods. Furthermore, because shape, color, and motion are decoupled in the latent space, the system supports "Cross Identity Transfer"—applying the motion of one sequence to the identity of another.

While the model performs exceptionally well, the authors note that future work will focus on overcoming data scarcity in high-quality 4D datasets and incorporating even stronger generative priors to enhance the realism of complex, multi-object interactions.

Summary updated

Abstract

Directly learning to model 4D content, including shape, color and motion, is challenging. Existing methods depend on skeleton-based motion control and offer limited continuity in detail. To address this, we propose a novel framework that generates coherent 4D sequences with animation of 3D shapes under given conditions with dynamic evolution of shape and color over time through integrative latent mapping. We first employ an integrative latent unified representation to encode shape and color information of each detailed 3D geometry frame. The proposed skeleton-free latent 4D sequence joint representation allows us to leverage diffusion models in a low-dimensional space to control the generation of 4D sequences. Finally, temporally coherent 4D sequences are generated conforming well to the input images and text prompts. Extensive experiments on the ShapeNet, 3DBiCar and DeformingThings4D datasets for several tasks demonstrate that our method effectively learns to generate quality 3D shapes with color and 4D mesh animations, improving over the current state-of-the-art. Source code will be released.
BibTeX
@misc{yang2024beyond,
  author = {Qitong Yang and Mingtao Feng and Zijie Wu and Shijie Sun and Weisheng Dong and Yaonan Wang and Ajmal Mian},
  title = {{Beyond Skeletons: Integrative Latent Mapping for Coherent 4D Sequence Generation}},
  year = {2024},
  eprint = {2403.13238},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2403.13238},
}
Figure