Abstract
We propose Pulsar, an efficient sphere-based differentiable rendering module that is orders of magnitude faster than competing techniques, modular, and easy-to-use due to its tight integration with PyTorch. Differentiable rendering is the foundation for modern neural rendering approaches, since it enables end-to-end training of 3D scene representations from image observations. However, gradient-based optimization of neural mesh, voxel, or function representations suffers from multiple challenges, i.e., topological in-consistencies, high memory footprints, or slow rendering speeds. To alleviate these problems, Pulsar employs: 1) a sphere-based scene representation, 2) a modular, efficient differentiable projection operation, and 3) (optional) neural shading. Pulsar executes orders of magnitude faster than existing techniques and allows real-time rendering and optimization of representations with millions of spheres. Using spheres for the scene representation, unprecedented speed is obtained while avoiding topology problems. Pulsar is fully differentiable and thus enables a plethora of applications, ranging from 3D reconstruction to neural rendering.
Introduction
In the field of "inverse graphics," the goal is to recover 3D scene properties—such as geometry, texture, and lighting—directly from 2D image observations. This process relies heavily on differentiable rendering, which acts as a bridge between the 3D representation and the 2D pixel output. By ensuring the rendering process is differentiable, gradients can flow from an image-based loss function back to the 3D parameters, enabling end-to-end optimization.
However, existing differentiable renderers face significant trade-offs. Meshes are efficient but struggle with topological changes during optimization. Voxels suffer from cubic memory scaling, limiting resolution. Implicit functions (like NeRF) offer high fidelity but are computationally expensive to render. Point-based methods are flexible but often lack the efficiency or the "soft" blending required for stable gradient-based optimization. Pulsar addresses these challenges by introducing a high-performance, sphere-based rendering framework.
The Pulsar Framework
Pulsar is built on three core pillars designed for speed and flexibility:
1. Sphere-Based Representation
Pulsar represents a 3D scene as a collection of $M$ spheres. Each sphere is defined by its 3D position, radius, opacity, and a "payload"—which can be an RGB color or a latent feature vector for neural shading. Spheres are mathematically efficient to intersect and avoid the connectivity constraints of meshes, allowing for seamless geometry optimization.
2. Differentiable Projection and Blending
To render a pixel, Pulsar identifies spheres intersecting the corresponding camera ray. It employs a weighted softmax blending function based on the spheres' depth. A critical innovation is the inclusion of a spatial distance factor ($d_i$) in the weight calculation. This ensures that spheres near a ray—even if not directly intersecting its center—contribute to the gradient, effectively "pulling" geometry toward the correct position during optimization.
3. Tile-Based CUDA Acceleration
Implemented as a custom PyTorch extension, Pulsar utilizes a tile-based approach to parallelize rendering across the image. A key feature is early stopping: once a pixel reaches a certain opacity threshold, the renderer stops processing further spheres along that ray. This leads to massive speedups and allows the system to handle millions of primitives with sub-linear scaling.
Performance and Results
Pulsar demonstrates remarkable performance gains over existing state-of-the-art differentiable renderers. In benchmarks involving ~233k points, Pulsar's CUDA backend achieves a forward pass in just 9ms, compared to 112ms for PyTorch3D and over 5000ms for Soft Rasterizer.
High-Fidelity Reconstruction
The system's efficiency translates directly into faster optimization cycles. In a 3D head reconstruction task, Pulsar reached convergence in just 31 seconds, whereas previous methods like DSS required nearly 20 minutes. Pulsar can reconstruct detailed models, including complex features like hair, using 400k spheres in under 75 seconds. The paper also validates Pulsar across diverse datasets, including the SoftRas Airplane for silhouette-based reconstruction and high-resolution NeRF scenes (Flower, Fern, Chair) for viewpoint synthesis.
Real-Time Neural Rendering
By decoupling geometry from shading, Pulsar can output latent feature maps that are subsequently processed by a neural network (e.g., pix2pixHD). This "neural shading" allows the model to learn complex visual effects like specular highlights or transparency without explicit physical modeling, all while maintaining 30+ FPS on consumer-grade hardware.
Conclusion
Pulsar represents a significant step forward in differentiable rendering, offering a solution that combines the flexibility of point-based methods with unprecedented speed and memory efficiency. While challenges remain—such as handling spheres smaller than a pixel and the specialized nature of its CUDA kernels—Pulsar provides a powerful tool for real-time 3D reconstruction and high-fidelity neural rendering.
Summary updated