When we capture the 3D world using LiDAR, depth cameras, or multi-view stereo, we naturally record data as a collection of discrete samples: a point cloud. While computationally efficient and flexible, representing a continuous world with zero-dimensional points introduces a fundamental geometric flaw known as the surface continuity gap.
If you look at a point cloud from the exact perspective it was captured, it appears solid. But as soon as you move the camera, change the scale, or warp the viewpoint, the illusion shatters. The underlying surface breaks down into a sparse collection of disconnected coordinates. Understanding why these artifacts happen—and how modern graphics pipelines patch them to build continuous surfaces—is central to modern 3D vision.
The Anatomy of a Hole
Why do sparse point clouds fail when rendered? The issue stems from how discrete elements behave under perspective transformations.
Captured View:
* * * * * (Points appear dense and continuous)
\ | | | /
\ | | | /
Virtual Camera: \ /
v v
Novel View:
* * * * * (Points spread apart, creating holes)
When a novel view is synthesized, image features or 3D points are projected onto a new image plane via forward-warping or splatting. As shown by Jia Zheng et al. in Layout-Guided Novel View Synthesis From a Single Indoor Panorama, this process stretches discrete pixels across larger target areas. Because the representation lacks inherent connectivity, this stretching leaves empty gaps, creating a sparse distribution characterized by numerous small holes.
This degradation worsens under specific conditions:
- Wide Baselines: When the distance between the original camera position and the target viewpoint is large, David Li et al. demonstrate in OmniSyn: Synthesizing 360 Videos with Wide-baseline Panoramas that projected points spread too far apart, causing severe see-through artifacts and a total loss of visual coherence (Minye Wu et al. in Multi-View Neural Human Rendering).
- Object Boundaries: At the silhouettes of foreground objects, the depth jumps sharply. Tatsuki Nagano et al. show in See-Through System Using LiDAR for Tele-Operated Construction Machines that these large spatial gaps cause severe texture stretching, smearing, and background bleeding.
- Scale Changes: If a user zooms in on a point cloud, the spatial density drops relative to the screen resolution. As noted by Ingmar Posner et al. (Efficient Non-Parametric Surface Representations Using Active Sampling for Push Broom Laser Data) and Mike Smith et al. (Adaptive Data Compression for Robot Perception), a discrete point cloud only provides information at the exact coordinates of a sensor strike. It cannot be queried at arbitrary densities between those strikes.
To overcome this, researchers have developed three core strategies: continuous splatting kernels, local neural fields, and alternative continuous geometries.
Strategy 1: Continuous Splatting Kernels
The most direct way to close the surface continuity gap is to replace zero-dimensional points with continuous, differentiable spatial distributions. Instead of projecting a single pixel coordinate, the pipeline projects a smooth mathematical function that can expand or contract to fill empty space.
Yueyu Hu et al. leverage this in Low Latency Point Cloud Rendering with Learned Splatting by predicting 3D elliptical Gaussian parameters for every point in a scene. Because Gaussians have continuous spatial extent, they naturally overlap and blend with their neighbors, producing hole-free, smooth surfaces during rasterization.
Discrete Points: . . . .
Gaussian Kernels: (***) (***) (***) (***)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Continuous, overlapping coverage
When dealing with highly sparse or non-uniform data, static kernels are not enough. To prevent semi-transparent artifacts and noisy geometry, Changfeng Ma et al. introduce a dynamic approach in Sparse Point Cloud Patches Rendering via Splitting 2D Gaussians, where 2D Gaussians are adaptively predicted and "split" to fill geometric gaps on the fly.
By coupling point-wise Gaussian regression with neural feature splatting, frameworks like the one proposed by Jiaxu Wang et al. (PFGS: High Fidelity Point Cloud Rendering via Feature Splatting) ensure full surface coverage during 2D projection, eliminating the surface bleeding and aliasing common to traditional point graphics. This spatial smoothness constraint is critical; as demonstrated by Luca Savant Aira et al. in Gaussian Splatting for Efficient Satellite Image Photogrammetry, without these continuous constraints, models trained on sparse data inevitably collapse into noisy, disjointed representations.
Strategy 2: Local Neural Fields and Implicit Manifolds
Rather than smoothing the points during rendering, another approach anchors continuous mathematical functions directly to the points in 3D space. This treats the point cloud not as the surface itself, but as a "scaffold" supporting a continuous field.
In Neural Points: Point Cloud Representation with Neural Fields for Arbitrary Upsampling, Wanquan Feng et al. redefine the point cloud paradigm by treating each point as the center of a local neural field. When the surface is magnified, the pipeline queries these local fields rather than interpolating between raw coordinates. This allows for arbitrary-factored upsampling while maintaining a perfectly smooth underlying manifold.
Point Scaffold: o o o
\ / \ /
Local Fields: [--neural field--] [--neural field--]
=======================================
Continuous Implicit Surface Manifold
Similarly, Shi-Lin Liu et al. combine the computational efficiency of points with the topological consistency of implicit functions in Deep Implicit Moving Least-Squares Functions for 3D Reconstruction. They use a deep network to generate a scaffold of oriented points that collectively define a global, continuous surface via Implicit Moving Least-Squares (IMLS).
When point density is insufficient, neural feature interpolation can still break down and cause localized spherical distortions on the surface (Shih-Yang Su et al. in NPC: Neural Point Characters from Video). To resolve this, Yiping Xie et al. develop DiffPBR: Point-Based Rendering via Spatial-Aware Residual Diffusion. Instead of relying purely on geometric interpolation, they splat 3D-consistent noise directly from the point cloud and use a spatial-aware residual diffusion process to seamlessly patch holes, ensuring photorealistic and view-consistent surface generation.
Strategy 3: Alternative Continuous Geometries
A third school of thought argues that to truly solve discretization artifacts, we must abandon point clouds and voxel grids entirely in favor of primitives with explicit or implicit continuity.
Historically, volumetric methods like binary occupancy grids or voxels have been used to represent 3D structures, but they suffer from severe cubic scaling laws and produce blocky, jagged approximations (Edmond Boyer et al. in On Computing Exact Visual Hulls of Solids Bounded by Smooth Surfaces). As explained by the authors of Octree Generating Networks: Efficient Convolutional Architectures for High-resolution 3D Outputs, binary voxel representations only identify the presence or absence of geometry, failing to capture rich, continuous surface details. This has driven a shift toward Signed Distance Functions (SDFs), as seen in Locally Attentional SDF Diffusion for Controllable 3D Shape Generation, which map continuous mathematical boundaries to capture smooth, high-resolution geometry.
For explicit rendering, researchers have introduced continuous geometric primitives that bridge the gap between flexible points and rigid meshes:
- Deformable Sheets: In Worldsheet: Wrapping the World in a 3D Sheet for View Synthesis from a Single Image, Nikhila Ravi et al. avoid point-cloud holes by "shrink-wrapping" a continuous, deformable mesh sheet onto an input image. Because the sheet maintains its topological connectivity during view synthesis, it completely eliminates the possibility of sparse gaps.
- Alpha Tablets: To capture sharp, irregular boundaries that point clouds struggle to delineate, Yushi Bai et al. introduce AlphaTablets: A Generic Plane Representation for 3D Planar Reconstruction from Monocular Videos. These are 3D rectangles equipped with alpha channels that combine the flexibility of discrete point-based optimization with the clean, continuous boundaries of planar geometry.
By moving from discrete coordinates to continuous representations—whether through overlapping Gaussian splats, localized neural fields, or deformable geometric sheets—modern 3D vision pipelines effectively bridge the surface continuity gap, turning sparse sensor data into watertight, photorealistic digital worlds.
Key Foundational Work
Efficient Non-Parametric Surface Representations Using Active Sampling for Push Broom Laser Data (2010) — contributed 1 concept elements to this direction.
On Computing Exact Visual Hulls of Solids Bounded by Smooth Surfaces (2001) — contributed 1 concept elements to this direction.