TL;DR
This paper introduces a novel system for robustly and autonomously tracking volumetric video sequences, even those with noisy or incomplete geometry from sparse capture setups. By leveraging shape similarity for keyframe selection and a novel correspondence framework, the system enables efficient compression, geometry recovery, and seamless propagation of user edits, making volumetric video more accessible.
What Problem Does This Paper Address?
Volumetric video, which captures and reconstructs 3D human performances, is gaining traction in industry and creative fields. However, creating high-quality volumetric content often requires expensive, multi-camera setups and computationally intensive per-frame 3D reconstruction. This results in large data sizes and challenges in maintaining temporal coherence, especially when dealing with noisy data, missing geometry (e.g., occluded limbs), or significant topological changes. Existing methods often struggle with these issues, requiring strong priors or failing on less ideal capture conditions. This paper aims to bridge this gap by providing an autonomous, robust tracking system that can handle these challenges, reduce data overheads, and allow for greater creative freedom through user edits.
Methods
The proposed system tackles the problem in several key stages:
- Abstraction Layer Creation: To handle noisy and dense input meshes, an abstraction layer is created by reducing the mesh complexity and applying adaptive isotropic remeshing. This provides a cleaner, more manageable representation for subsequent steps.
- Similarity-Driven Automatic Keyframe Selection: Instead of relying on heuristics that fail with noisy data, the system uses spherical harmonics to compute rotation-invariant shape descriptors. A similarity score is derived from these descriptors, which helps partition the sequence into regions. Keyframes within these regions are selected based on minimizing cumulative tracking error, ensuring a minimal set of frames captures the sequence's motion and shape.
- Dense Volumetric Correspondences: The system establishes dense correspondences between abstraction layers by volumetrically segmenting them. This segmentation is transferred from the source to the target abstraction layer. A modified Coherent Point Drift (CPD) algorithm is used for segment-wise alignment, with special handling for missing geometry (using rigid ICP) to preserve structure.
- Deformation Graph Construction and Application: A deformation graph is built from the aligned abstraction layers. This graph represents the mesh as a set of nodes with associated rotations and translations, allowing for smooth, non-rigid deformation. The correspondences derived earlier drive the optimization of this graph.
- Detail Synthesis: To address popping artifacts at region boundaries caused by keyframing, a boundary-blending interpolation technique is employed. This creates a smoother transition by synthesizing surface details.
- Sequence Smoothing: A standard 3D Kalman filter is applied to the vertex positions to reduce high-frequency temporal noise (flickering) and improve visual quality, especially for recovered geometry and user edits.
Datasets
The paper evaluates its method on several datasets:
- Ground-truth sequences: These are described as sequences generated by animating a pre-defined template, considered free from reconstruction artifacts, allowing for objective evaluation using metrics like Hausdorff distance. Specific sequences mentioned include "Crane," "Jumping," "Bouncing," "Handstand," and "Swing."
- Challenging sequences with drastic topology changes and fast motion: These are captured with sparse multi-view setups (e.g., a budget studio with 12 RGB cameras) and are characterized by noise and occlusion.
- Volumetric data captured from multi-view capture setups: This includes sequences with significant occlusion and fast-moving objects, like a football being volleyed.
The paper does not explicitly mention introducing new benchmarks but rather uses existing datasets for evaluation.
Key Insights and Ideas
- Shape similarity as a robust keyframing metric: Using spherical harmonics for shape descriptors provides a powerful way to identify significant changes in a sequence, even with noisy input, leading to more efficient and accurate keyframe selection.
- Abstraction layers for generality: Reducing mesh complexity via abstraction layers makes the correspondence and registration process more robust and computationally efficient, regardless of the input mesh density.
- Volumetric correspondences over normal-constrained ICP: The use of volumetric correspondences, derived from segmentation, allows for more reliable matching in challenging scenarios with fast motion and significant pose changes, outperforming methods relying solely on normal-constrained ICP.
- Autonomous geometry recovery and edit propagation: The system's ability to detect and recover missing geometry, and to seamlessly propagate user-defined edits across frames, significantly enhances creative control and workflow efficiency for volumetric content creation.
- Boundary-aware detail synthesis: This technique effectively mitigates popping artifacts at keyframe boundaries, leading to smoother visual transitions, especially in sequences with large topological changes.
Key Results
- Keyframing: The proposed keyframe selection method significantly reduces the number of keyframes needed (e.g., 13 vs. 19 in a "twirl" sequence) while achieving smaller errors (e.g., Max Error of 0.0651 vs. 0.0662 for Collet et al. [4]).
- Tracking Accuracy: The system demonstrates superior performance on ground-truth sequences, achieving lower Maximum and Median Hausdorff Errors compared to state-of-the-art methods like Guo et al. [9] and Amberg et al. [1]. For instance, on the "Crane" sequence, the Max Error is 0.0424 compared to 0.3432 and 0.3753.
- Qualitative Results: The system shows robustness in handling fast motion, large deformations, and persistent geometry challenges (e.g., recovering a missing leg, tracking a fast-moving football). It also effectively propagates user edits, even complex ones like adding props.
- Detail Synthesis: The proposed method produces smoother transitions across frames compared to existing techniques, avoiding sharp boundary transitions and noticeable popping effects.
Limitations and Future Work
- Limitations:
- While the system is autonomous, it still requires keyframes, which represent a workload.
- The Kalman filter, while effective, can sometimes inhibit motion over time, leading to noticeable popping effects, which the authors attempt to mitigate with an adaptive approach.
- The system's current approach to handling missing geometry involves rigid ICP, which might not be ideal for all scenarios.
- Future Work:
- Extend the work to create a global template without relying on parametric templates or pre-defined animation rigs.
- Further explore and improve the 3D Kalman smoothing technique, investigating its many evolutions.
- The authors hope their approach will inspire further work towards low-end, cost-effective volumetric video solutions, making the medium more accessible to a wider range of users.
Summary updated