Mixture of volumetric primitives for efficient neural rendering

Stephen Lombardi, Tomas Simon, Gabriel Schwartz, Michael Zollhöfer, Yaser Sheikh, Jason M. Saragih

Introduction

Creating realistic, dynamic 3D human avatars in real-time remains a significant challenge in computer graphics, with applications ranging from telepresence to immersive gaming. Existing neural rendering techniques often face a trade-off between speed, memory efficiency, and visual fidelity. Traditional triangle meshes struggle with complex, "fuzzy" structures like hair and topological changes. Neural volumes (voxels) are memory-intensive, scaling cubically ($O(n^3)$) and often resulting in low-resolution outputs. Meanwhile, Neural Radiance Fields (NeRF) provide high detail but are computationally expensive, requiring thousands of network evaluations per pixel.

To address these limitations, this work introduces Mixture of Volumetric Primitives (MVP), a hybrid representation that combines the flexibility of volumetric rendering with the efficiency of primitive-based methods.

The MVP Representation

The core of MVP is the decomposition of a scene into a collection of small, local volumetric primitives. Each primitive is a voxel grid containing color and opacity data. Unlike static global volumes, these primitives are dynamic; they can move, rotate, and scale to follow the subject's motion.

The evolution of the MVP representation from a single global volume (1 prim.) to thousands of local primitives (4k prim.). Increasing the number of primitives reduces unnecessary sampling in empty space and improves reconstruction quality.

Efficient Architecture and Rendering

Instead of using a slow Multi-Layer Perceptron (MLP) for every point in space, MVP utilizes a 2D convolutional neural network (CNN) to generate voxel data for all primitives simultaneously. This "spatial sharing" significantly reduces computational overhead.

The rendering process is optimized through:

  • Bounding Volume Hierarchy (BVH): By only sampling points inside active primitives and using a BVH to skip empty space, the system achieves frame rates orders of magnitude faster than standard volumetric methods.
  • Opacity Fade Factor: A windowing function is applied to the edges of each primitive. This encourages the model to represent the scene through the motion of the boxes rather than static "painting" at the boundaries, leading to cleaner reconstructions of complex features like flyaway hairs.
A visualization of the volumetric primitives (left) and the resulting rendered avatar (right). The primitives adaptively cover the geometry to maximize efficiency.

Experimental Results

The researchers evaluated MVP using a high-end multi-camera capture system with approximately 100 synchronized cameras. The model was trained on roughly 25,000 frames per subject and tested on novel view synthesis.

Key findings include:

  • Performance: MVP is approximately 1000x faster than NeRF, achieving real-time performance of 20–40 FPS on modern hardware.
  • Quality: The method produces sharper results than previous volumetric approaches, particularly in challenging areas like the interior of the mouth and fine hair structures.
  • Scalability: By finding a "sweet spot" (typically between 256 and 1024 primitives), MVP balances memory usage and rendering speed effectively. A configuration with 256 primitives and 8 million total voxels achieves a high PSNR of ~32.4.

Conclusion and Future Work

MVP represents a significant step toward efficient, high-quality neural rendering of dynamic subjects. By hybridizing volumetric and primitive-based approaches, it overcomes the resolution limits of voxels and the speed bottlenecks of NeRF.

Current limitations include a dependency on a coarse tracked mesh for initialization and potential performance drops when many primitives overlap. Future research will focus on allowing primitives to "self-organize" directly from images and further optimizing the primitive distribution to handle complex occlusions more efficiently.

Summary updated

Abstract

Real-time rendering and animation of humans is a core function in games, movies, and telepresence applications. Existing methods have a number of drawbacks we aim to address with our work. Triangle meshes have difficulty modeling thin structures like hair, volumetric representations like Neural Volumes are too low-resolution given a reasonable memory budget, and high-resolution implicit representations like Neural Radiance Fields are too slow for use in real-time applications. We present Mixture of Volumetric Primitives (MVP), a representation for rendering dynamic 3D content that combines the completeness of volumetric representations with the efficiency of primitive-based rendering, e.g., point-based or mesh-based methods. Our approach achieves this by leveraging spatially shared computation with a convolutional architecture and by minimizing computation in empty regions of space with volumetric primitives that can move to cover only occupied regions. Our parameterization supports the integration of correspondence and tracking constraints, while being robust to areas where classical tracking fails, such as around thin or translucent structures and areas with large topological variability. MVP is a hybrid that generalizes both volumetric and primitive-based representations. Through a series of extensive experiments we demonstrate that it inherits the strengths of each, while avoiding many of their limitations. We also compare our approach to several state-of-the-art methods and demonstrate that MVP produces superior results in terms of quality and runtime performance.
BibTeX
@article{lombardi2021mixture,
  author = {Stephen Lombardi and Tomas Simon and Gabriel Schwartz and Michael Zollhöfer and Yaser Sheikh and Jason M. Saragih},
  title = {{Mixture of volumetric primitives for efficient neural rendering}},
  journal = {ACM Trans. Graphics},
  year = {2021},
  doi = {10.1145/3450626.3459863},
}
Figure