Abstract
RePerformer is a novel framework designed to bridge the gap between high-fidelity volumetric video playback and flexible reperformance. While traditional methods either offer photorealism without motion control or motion control with limited detail, RePerformer achieves both. By disentangling dynamic scenes into hierarchical 3D Gaussians and employing a Morton-based 2D mapping strategy, the system enables photorealistic rendering of complex human-object interactions under novel poses. Experimental results demonstrate superior generalization and visual quality compared to existing state-of-the-art methods, achieving a PSNR of 30.06 on novel motion benchmarks.
Introduction
Volumetric video technology allows users to view moving scenes from any angle, providing an immersive experience. However, current solutions typically fall into two restrictive categories: replay-only videos that offer high detail but are static in motion, and animatable avatars that allow for new motions but struggle with complex geometry like loose clothing or musical instruments.
RePerformer introduces a "playback-then-reperformance" paradigm. It allows a high-fidelity recording of a complex scene—such as a musician playing an instrument—to be realistically animated with entirely new, unseen motions while maintaining the original visual fidelity.
Methodology
The RePerformer framework utilizes 3D Gaussian Splatting (3DGS) as its core representation and is built upon three technical pillars:
Hierarchical Disentanglement
To manage complex dynamics, the scene is split into two layers: Motion Gaussians, a sparse set of control points that capture underlying movement, and Appearance Gaussians, a dense set of hundreds of thousands of Gaussians that provide fine-grained visual detail.
Morton-based 3D-to-2D Mapping
A key challenge in animating 3D Gaussians is ensuring generalization to new poses. RePerformer addresses this by projecting 3D Appearance Gaussians onto 2D maps using a Morton-order (Z-order curve) parameterization. This mapping preserves spatial locality, allowing 3D attributes (color, opacity, rotation) to be packed into 2D "attribute maps."
By converting the problem into a 2D domain, the system can leverage standard 2D U-Nets with self-attention to learn the relationship between a pose (position map) and its appearance. This approach generalizes significantly better to new motions than traditional frame-by-frame 3D optimization.
Semantic-Aware Re-performance
To transfer motion from a driver sequence to a target subject, the system employs a semantic alignment module. Using tools like Language-SAM and GroundingDINO, RePerformer automatically identifies body parts and objects, ensuring contextually accurate motion transfer even between performers with different body proportions.
Results and Discussion
The researchers evaluated RePerformer using the DualGS dataset, which contains complex interactions involving violins, basses, and flutes.
- Generalization: On novel motions unseen during training, RePerformer achieved a PSNR of 30.06, significantly outperforming competitors such as AP-NeRF (26.85) and SC-GS (19.18).
- Efficiency: By relying on 2D CNNs for attribute prediction rather than exhaustive 3D per-frame optimization, RePerformer offers a more efficient training pipeline compared to high-fidelity alternatives like DualGS.
- Visual Fidelity: Qualitative analysis shows that the model maintains sharp details on thin structures (e.g., violin bows) and complex textures that typically cause artifacts in mesh-based or NeRF-based avatars.
Conclusion
RePerformer represents a significant step forward in volumetric video by unifying high-fidelity playback with photorealistic reperformance. By leveraging Morton mapping to bridge 3D representations with 2D convolutional power, it overcomes the limitations of rigid human templates. Future work will focus on improving motion tracking speeds and expanding the diversity of motions the system can generalize toward.