Abstract
Modeling and rendering of dynamic scenes is challenging, as natural scenes often contain complex phenomena such as thin structures, evolving topology, translucency, scattering, occlusion, and biological motion. Mesh-based reconstruction and tracking often fail in these cases, and other approaches (e.g., light field video) typically rely on constrained viewing conditions, which limit interactivity. We circumvent these difficulties by presenting a learning-based approach to representing dynamic objects inspired by the integral projection model used in tomographic imaging. The approach is supervised directly from 2D images in a multi-view capture setting and does not require explicit reconstruction or tracking of the object. Our method has two primary components: an encoder-decoder network that transforms input images into a 3D volume representation, and a differentiable ray-marching operation that enables end-to-end training. By virtue of its 3D representation, our construction extrapolates better to novel viewpoints compared to screen-space rendering techniques. The encoder-decoder architecture learns a latent representation of a dynamic scene that enables us to produce novel content sequences not seen during training. To overcome memory limitations of voxel-based representations, we learn a dynamic irregular grid structure implemented with a warp field during ray-marching. This structure greatly improves the apparent resolution and reduces grid-like artifacts and jagged motion. Finally, we demonstrate how to incorporate surface-based representations into our volumetric-learning framework for applications where the highest resolution is required, using facial performance capture as a case in point.
Introduction
Creating digital doubles of humans or objects usually relies on polygon meshes. While meshes are effective for solid surfaces, they struggle with the "messy" parts of the real world: thin structures like hair, evolving topologies like billowing smoke, and semi-transparent materials. If one attempts to track a person running their hands through their hair using a mesh, the geometry often collapses or glitches.
Existing alternatives like light field video allow for high-quality rendering but usually require the viewer to stay within a very narrow range of motion. Neural Volumes addresses these limitations by providing a representation that is 3D-aware (allowing free camera movement), end-to-end trainable (learning directly from images), and capable of handling complex, non-solid geometry.
Methods
The Neural Volumes pipeline consists of three main stages designed to transform multi-view video into a renderable 3D volume.
- The Encoder-Decoder: The system takes images from a subset of cameras and passes them through a Convolutional Neural Network (CNN) to create a low-dimensional latent code ($z$). This code represents the state of the scene at that specific moment. A decoder then transforms this code into a 3D voxel grid containing color (RGB) and opacity ($\alpha$).
- The Warp Field: Voxel grids are memory-intensive; doubling the resolution increases memory usage eightfold. To bypass this, the authors learn a mixture of affine warps that maps the output space to a "template" volume. This allows the network to "stretch" its resolution where it's needed most (like the fine details of a face) and ignore empty space, effectively simulating a high-resolution non-uniform grid.
- Differentiable Ray Marching: To train the network, the system compares its 3D volume to the original 2D images. A differentiable ray-marching algorithm simulates how light travels through the semi-transparent volume. Because the whole process is differentiable, the error between the rendered image and the ground truth can be backpropagated to update the neural network's weights.
Datasets and Evaluation
The authors captured high-quality data using a custom multi-view rig consisting of 34 synchronized 4K color cameras arranged in a hemisphere. Subjects included a variety of challenging targets such as moving hands, swinging hair, dry ice smoke, and facial performances. The method was benchmarked against scenes from publicly available Multi-View Stereo (MVS) datasets and traditional reconstruction tools like COLMAP and Agisoft Metashape.
Results and Discussion
Neural Volumes demonstrates significant advantages over traditional mesh-based or MVS methods, particularly for complex phenomena.
Key Insights
- Optimization Benefits: Unlike meshes, which have local support, semi-transparent volumes disperse gradient information along the entire ray of light. This creates a wider "basin of convergence," making it easier for the model to find a good geometric solution.
- Resolution Efficiency: By using an inverse warp field, the model can represent fine details in the output space that are actually larger in the template space, effectively overcoming the resolution limits of a standard $128^3$ grid.
- View-Conditioning: By feeding the camera's viewing angle into the decoder, the model learns to render specularities—the way light glints off a surface—which are inherently view-dependent.
Performance
Neural Volumes successfully reconstructed "un-meshable" phenomena like the fine tendrils of dry ice smoke and the fuzzy texture of a stuffed toy. Traditional MVS methods produced jagged, noisy, or incomplete geometry on these same subjects. Quantitatively, the "warp-space affine mixture" model significantly outperformed standard voxel grids and convolutional warp models in Mean Squared Error (MSE) during validation on unseen camera views.
Limitations and Future Work
Despite its success, several challenges remain. Memory constraints continue to be a bottleneck due to the cubic relationship between resolution and memory. The current model handles "dull" specularities well but struggles with highly refractive objects (like glass) or mirror-like reflections. Additionally, the system does not explicitly model temporal consistency; moving through the latent space may produce visually correct but physically impossible animations. Finally, the system works best when the object is isolated, as complex static backgrounds can sometimes interfere with the reconstruction.
Summary updated