Abstract
Directly learning to model 4D content, including shape, color, and motion, is challenging. Existing methods depend on skeleton-based motion control and offer limited continuity in detail. To address this, we propose a novel framework that generates coherent 4D sequences with animation of 3D shapes under given conditions with dynamic evolution of shape and color over time through integrative latent mapping. We first employ an integrative latent unified representation to encode shape and color information of each detailed 3D geometry frame. The proposed skeleton-free latent 4D sequence joint representation allows us to leverage diffusion models in a low-dimensional space to control the generation of 4D sequences. Finally, temporally coherent 4D sequences are generated conforming well to the input images and text prompts.
Introduction
Generating 4D content—3D geometry that evolves over time—is a massive challenge in computer vision. Current approaches usually fall into two camps: prior-based methods that rely on restrictive skeletal rigs (like SMPL), and free-form methods (like NeRF or Gaussian Splatting) that often suffer from temporal flickering or a lack of underlying geometric structure. This paper identifies a "representation gap" and proposes a solution that enables efficient spatio-temporal diffusion to produce coherent animations without relying on pre-defined skeletons.
Methodology
The proposed framework bridges the representation gap through a three-stage approach:
1. Coherent 3D Shape and Color Modeling
Instead of treating a 3D frame as a single blob of data, the system decouples it into a Signed Distance Function (SDF) for shape and a Continuous Color Field for appearance. A 3D U-Net refines the coarse SDF to capture fine details. This decoupling allows for independent editing, such as changing a character's clothes without altering its shape.
2. Matrixized 4D Representation
To handle the temporal dimension without exploding computational costs, the authors concatenate the latent vectors of keyframes into a matrix $\mathcal{M}$. Intermediate frames are recovered via latent space interpolation. This "matrixized" approach captures the 4D distribution of a sequence more efficiently than processing every frame individually, reducing complexity from $O(T^2)$ to $O(\frac{T^2}{K^2})$.
3. Hierarchical Conditional Spatio-Temporal Attention (HCSTA)
A specialized diffusion block uses a staged approach to ensure consistency:
- Shape-Color Cross-Attention: Ensures the color "sticks" to the geometry.
- Conditioned Cross-Attention: Injects the text and image prompts.
- Temporal Self-Attention: Enforces consistency across the timeline.
Experimental Results
The model was evaluated on diverse benchmarks including ShapeNet, 3DBiCar, and DeformingThings4D (DT4D). The results demonstrate significant improvements in both speed and quality:
- Generation Speed: The model generates a 4D sequence in approximately 0.625 seconds, significantly faster than DreamGaussian4D (~6.5 minutes) or STAG4D (~7.5 minutes).
- Quantitative Superiority: On the 3DBiCar and DT4D datasets, the method achieved the best (lowest) Minimum Matching Distance (MMD) and highest Coverage (COV), indicating more realistic and diverse generations.
- Qualitative Fidelity: User studies rated the model higher than competitors in terms of spatio-temporal consistency and alignment with input prompts, producing smoother surfaces and more detailed features.
Key Insights and Discussion
A critical advantage of this method is Geometry-Aware Color. By linking the color field to SDF values, the model ensures color remains consistent even as the object deforms, preventing the "color drifting" common in other methods. Furthermore, because shape, color, and motion are decoupled in the latent space, the system supports "Cross Identity Transfer"—applying the motion of one sequence to the identity of another.
While the model performs exceptionally well, the authors note that future work will focus on overcoming data scarcity in high-quality 4D datasets and incorporating even stronger generative priors to enhance the realism of complex, multi-object interactions.
Summary updated