Abstract
Synthesizing novel views of moving humans requires maintaining consistency across both camera rotations and time. We present a deep learning framework that utilizes a siamese network architecture and a 3D-aware gating mechanism to generate high-quality, temporally and rotationally consistent human performance videos. By decomposing 3D gating into 2D spatial planes and enforcing bi-directional consistency losses, our method significantly reduces artifacts and flickering compared to existing state-of-the-art approaches.
Introduction
Novel View Synthesis (NVS) aims to generate entirely new viewpoints of a subject from a limited set of input angles. While static object NVS has seen significant progress, human performance video introduces two major challenges: complex asymmetry and the consistency gap. Humans are highly deformable and asymmetric, making it easy for models to "hallucinate" incorrect geometry. Furthermore, treating frames or viewpoints in isolation leads to temporal flickering and spatial "popping" during camera rotation. This paper addresses these issues by creating a model that understands 3D structure and temporal flow simultaneously.
Methodology
The core of the proposed framework is a two-tower siamese network designed to process consecutive video frames ($t$ and $t-1$). The architecture is divided into four primary components:
- 3D-Aware Latent Representation: The model transforms 2D input pixels into a 3D volumetric feature space, allowing for mathematical rotation of the subject before rendering.
- Gated Volume Completion: To handle the memory constraints of 3D convolutions, the authors propose a Volume Gating layer that decomposes gating into three 2D spatial planes ($O(n^2)$). This helps the model distinguish foreground human features from background noise without the $O(n^3)$ cost of full 3D gating.
- Temporal Feature Augmentation: Features are shared between consecutive time steps using optical flow to warp features from the previous frame to the current one, preserving details that might be occluded in a single frame.
- Consistency Losses: Rotational and temporal consistency losses ensure that adjacent generated views and consecutive frames can be warped into one another, stabilizing the output.
Experimental Results
The model was evaluated on the newly introduced Multi-View Human Action (MVHA) dataset, which contains 30 high-quality 3D human models performing diverse motions across 54 viewpoints. It was also tested on the PVHM benchmark and ShapeNet.
Quantitative Performance
On the PVHM dataset, the proposed method achieved an MSE of 61.68 and an SSIM of 0.9807, outperforming the previous state-of-the-art (Transformable Bottleneck Networks), which scored 70.34 and 0.9695, respectively. Lower rotational and temporal loss scores confirmed that the generated videos were significantly smoother.
Qualitative Analysis
Visually, the model excels at reconstructing fine-grained details such as sharp limbs and realistic clothing wrinkles. While baseline models often suffer from "ghosting" or blurred extremities during complex motions like dancing, this gated volumetric approach maintains structural integrity across both time and rotation.
Conclusion and Future Work
This work demonstrates that 2D-decomposed 3D gating and bi-directional supervision are effective strategies for consistent human NVS. While the model sets new benchmarks, future work will focus on improving the handling of complex dynamic backgrounds and optimizing the architecture for real-time AR/VR applications. The success of the gated volumetric approach on ShapeNet also suggests that these techniques are robust for general 3D object synthesis beyond human performance.
Summary updated