Real Time View Synthesis with 3D Gaussian Splatting

141 concepts138 papers1 proposal

For years, the gold standard for neural scene reconstruction was dominated by continuous, coordinate-based representations like Neural Radiance Fields (NeRFs). NeRFs achieved breathtaking photorealism by encoding a scene's geometry and color within the weights of a multilayer perceptron (MLP). However, rendering a single pixel required marching a ray through space and querying this neural network dozens of times. This heavy computational tax made real-time interactive rendering an elusive goal on consumer hardware.

The field has undergone a massive paradigm shift toward explicit, point-based representations—most notably 3D Gaussian Splatting (3DGS). Instead of querying a continuous volume, 3DGS represents the world using millions of overlapping 3D ellipsoids (Gaussians). Each primitive is defined by explicit parameters: a 3D center position, an orientation, a scale matrix, an opacity value, and a set of spherical harmonics coefficients that capture view-dependent color.

The core intuition behind why 3D Gaussian Splatting works so fast lies in its compatibility with modern graphics hardware. Instead of performing expensive volumetric ray integration, the 3D Gaussians can be projected (or "splatted") onto a 2D image plane. The GPU can then sort these projected primitives and alpha-blend them in a highly parallelized, front-to-back rasterization pipeline. This bypasses the neural network bottleneck entirely during inference, unlocking rendering speeds of hundreds of frames per second.

Animating the Primitives: Digital Humans and Avatars

While static scenes benefit immensely from point-based rasterization, animating dynamic objects like human faces and bodies introduces a new layer of complexity. Early neural avatar methods relied on image-to-image translation networks to map sparse 3D body vertices directly to pixels, as seen in the work of Prokudin et al., or optimized ray sampling via empty-space skipping in dynamic NeRFs, as demonstrated by Jiang et al.. However, these methods still struggled to maintain interactive frame rates at high resolutions.

By anchoring explicit 3D Gaussians to parametric models, researchers found a way to combine structural control with rasterization speed. For instance, Qian rigged 3D Gaussians directly to the triangles of a FLAME parametric face model. When the underlying mesh deforms based on expression or pose coefficients, the attached Gaussians transform along with it, preserving high-fidelity details during animation. To push performance even further, Xiang et al. embedded a sparse set of Gaussians onto a parametric face model, achieving photorealistic head avatar rendering at 300 frames per second.

[Parametric Model (FLAME/SMPL)] ---> [Rigged 3D Gaussians] ---> [Tile-Based Rasterizer] ---> [Real-Time Avatar (300+ FPS)]

To completely eliminate the overhead of neural deformation networks during rendering, the field has embraced classical computer graphics mathematics. Zhou et al. integrated linear blendshape mathematics with point-based radiance fields. Because the properties of the Gaussians are calculated as simple linear combinations of base facial expressions, the animation step reduces to a series of fast matrix multiplications executed right before rasterization, enabling rendering speeds up to 370 FPS.

Capturing the Moving World from Casual Video

Reconstructing dynamic scenes from "in-the-wild" monocular video introduces a severe underconstrained problem: if both the camera and the subject are moving, how do we untangle camera trajectory from scene deformation?

Traditional pipelines rely heavily on Structure-from-Motion (SfM) preprocessing tools like COLMAP to calculate camera poses before optimization begins. However, SfM often fails when videos contain large camera motions, featureless regions, or significant subject movement. To bypass this brittle bottleneck, Yao et al. proposed a hierarchical training and merging framework that optimizes the scene directly from video frames without a global SfM step. Taking a different approach to motion modeling, Park et al. utilized motion-adaptive splines to jointly optimize camera poses and dynamic 3D Gaussians from a single monocular video source.

Traditional Pipeline:  [Casual Video] ---> [Brittle SfM / COLMAP] ---> [3DGS Optimization]
SfM-Free Pipeline:     [Casual Video] ---> [Joint Spline/Hierarchical Optimization] ---> [Dynamic 3DGS]

Furthermore, real-world casual captures are rarely pristine; they are frequently plagued by motion blur, defocus, and varying exposures. Point-based representations can be made robust to these artifacts by incorporating the physics of camera imaging into the optimization loop. For example, Gong et al. jointly optimized camera trajectories, exposure variations, and the camera response function within a unified imaging model. To handle spatial degradation, Zhang et al. integrated a per-pixel blur prediction network directly into the point-based rendering pipeline, allowing the system to optimize sharp 3D Gaussians even when trained on highly blurred monocular inputs.

Scaling Up and Generalizing

Standard 3DGS is an optimization-based method, meaning it must spend minutes or hours fitting a specific set of Gaussians to a single scene. To make this technology viable for large-scale environments or instant asset generation, the representation must either scale efficiently or transition into a feed-forward model.

For massive, unbounded environments like those found in driving simulations, memory footprint becomes the primary constraint. Reiser et al. demonstrated that combining sparse 3D grids with high-resolution 2D feature planes allows scenes to be compressed drastically, enabling real-time rendering directly within standard web browsers. When ground-truth sensor data like LiDAR is unavailable, vision-only approaches must infer the underlying geometry. To solve this, Li et al. paired self-supervised visual odometry with 3DGS to generate high-fidelity driving simulations from raw RGB streams alone.

The ultimate frontier is eliminating scene-specific optimization entirely. Feed-forward frameworks leverage the powerful geometric and contextual priors of pre-trained generative models to perform instant reconstruction. By utilizing multi-view diffusers equipped with epipolar attention mechanisms, models can directly predict consistent depth and appearance maps from a single image. As shown by Wu et al., these predicted maps can be explicitly unprojected into a 3D Gaussian representation in a matter of seconds, combining the imaginative power of generative AI with the lightning-fast rendering speeds of point-based rasterization.

History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_state_of_the_art_novel_view_synthesis_1b9492b2,
  title = {state-of-the-art novel view synthesis},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/state-of-the-art-novel-view-synthesis/rdr_1b9492b201f445e78c817bd226681535},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_state_of_the_art_novel_view_synthesis_1b9492b2_v1,
  title = {state-of-the-art novel view synthesis},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/state-of-the-art-novel-view-synthesis/rdr_1b9492b201f445e78c817bd226681535/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake