In 3D computer graphics and vision, representing geometry has traditionally forced a difficult compromise between explicit control and topological flexibility.
On one hand, polygon meshes offer highly efficient rendering and explicit surface tracking. However, they are bound by a fixed connectivity graph (a topology). If a surface undergoes drastic deformation—such as a character putting on a loose dress, a liquid splitting into droplets, or an object fracturing—a mesh template fails. It stretches unnaturally, tears, or requires complex, computationally expensive remeshing algorithms.
On the other hand, implicit volumetric fields (such as Signed Distance Functions or Neural Radiance Fields) handle arbitrary topological changes gracefully because they represent shapes as level sets of a continuous function. Yet, this flexibility comes at a steep price: animating, deforming, or rendering them requires dense volumetric sampling or expensive surface extraction steps like Marching Cubes.
To bridge this gap, a powerful paradigm has emerged: representing complex, deformable, and topology-changing surfaces as unstructured point clouds. By discarding the explicit edge connectivity of meshes while retaining the spatial localization of explicit representations, point-based geometries provide a highly flexible, template-free alternative for 3D modeling, generation, and simulation.
Breaking the Connectivity Constraint
The core intuition behind point-based surface modeling is simple: if you remove the edges connecting the vertices of a mesh, the geometric representation is no longer constrained by a fixed topology. Points can move independently, cluster together to form dense structures, or separate to represent disconnected components.
This flexibility is particularly evident in modeling human clothing. Traditional human avatars rely on parametric templates (like SMPL) that act as a skin-tight mesh. While this works well for tight garments, it fails for loose clothing like skirts or dresses, which do not share the topology of the human body.
Mesh Representation: (Point)---(Point)---(Point) --> Fixed edges stretch/tear under deformation
Point Representation: (Point) (Point) (Point) --> Points move freely, adapting to any topology
By transitioning to point clouds, Ilya Zakharkin et al. demonstrated that a single neural network can represent diverse garment types—ranging from tight shirts to flowing dresses—without requiring item-specific templates. Similarly, the authors of PocoLoco leveraged denoising diffusion on unordered point clouds to generate loose clothing, completely avoiding the mesh-tearing artifacts that plague parametric body models. Because the representation is decoupled from a fixed connectivity graph, garments that do not conform to the underlying body structure can be modeled with high fidelity.
How Deformable Point Fields Work
To animate or optimize an unstructured point cloud without losing the underlying shape's coherence, researchers typically employ a canonical deformation field.
Instead of predicting the 3D coordinates of a shape directly for every frame or pose, the geometry is defined as a dense set of points in a fixed, "canonical" space. When the object deforms, a learned neural network predicts displacement vectors or coordinate offsets for each individual point based on the target pose or frame.
[Canonical Space Points] ---> (Learned Deformation Field) ---> [Target Pose Surface]
For instance, Yufeng Zheng et al. utilized this canonical mapping to reconstruct highly detailed head avatars from videos. Because hair strands and accessories create complex, non-manifold geometries, explicit meshes fail to track them. By mapping unstructured points through a deformation field, their method retains the efficiency of explicit forward rendering while matching the topological fluidity of volumetric fields.
This mechanism also allows for independent point translations. As shown by Theo Deprelle et al., applying independent translation vectors to points in a set allows a model to learn elementary building blocks for shape generation, bypassing the constraints of deforming continuous 2D patches.
Resolution Independence in Generative Modeling
When training generative models (such as diffusion models or auto-encoders) to synthesize 3D shapes, traditional grid-based structures like voxels suffer from cubic computational scaling ($O(N^3)$), which severely limits their spatial resolution. Meshes avoid this but struggle to represent thin structures or complex topologies during optimization.
Unstructured point clouds solve this by acting as finite discrete sets that can scale efficiently. Christian Rupprecht et al. demonstrated that projection-conditioned point diffusion models can move individual points freely in 3D space to capture fine-grained geometric details and thin structures that are typically smoothed out or lost in discretized voxel volumes.
Furthermore, because point clouds do not enforce a specific manifold structure, they can be organized hierarchically. Lyne P. Tchapmi et al. introduced a structural decoder that models point clouds using a rooted tree structure. By assigning different branches of the tree to disconnected components of a shape, the network naturally learns to represent arbitrary topologies without being forced into a predefined geometric template.
Generalization Across Inconsistent Data Formats
In real-world applications, 3D data comes from highly inconsistent sources: high-resolution clean meshes from artists, noisy and partial point clouds from LiDAR, or fragmented depth maps from consumer RGB-D cameras.
Because meshes rely heavily on consistent vertex ordering and connectivity graphs, a model trained on one mesh topology cannot easily ingest data from another. Point-based representations discard this requirement entirely. By treating every input as an unstructured collection of spatial coordinates, models become agnostic to the underlying data format.
This format independence enables robust cross-dataset generalization:
- Correspondence Matching: Hyomin Kim et al. developed a virtual marker framework for articulated shapes that processes raw depth maps and full 3D meshes interchangeably, relying solely on the unstructured point distribution rather than mesh connectivity.
- Skeletonization: Zhiyang Dou et al. formulated 3D shape skeletonization as a set-cover problem over discrete surface points, allowing their method to extract clean skeletal structures from raw point clouds, polygon soups, and non-watertight meshes alike.
- Texture Mapping: Even when dealing with the "unruly" and fragmented geometries generated by Neural Radiance Fields (NeRFs), sampling points directly from the surface density allows neural UV mapping methods to create coherent texture atlases without relying on clean mesh manifolds (Pratul P. Srinivasan et al.).
Extending to Physics and Simulation
The advantages of point-surface representations extend far beyond static shape modeling and animation; they fundamentally simplify physical simulations and numerical analysis on complex geometries.
In traditional computational physics, solving partial differential equations (PDEs) on curved surfaces requires constructing a high-quality conformal mesh—a process that is notoriously prone to failure on intricate geometries. Mesh-free numerical methods, such as the Radial Basis Function (RBF)-Finite Difference method proposed by Varun Shankar et al., bypass this entirely. By utilizing scattered point clouds and local surface normals, they define differential operators directly on the points, achieving high-order convergence without ever establishing a connected mesh.
Similarly, in acoustic and dynamic simulations, point clouds offer a computationally lightweight alternative for modeling wave interactions and contact mechanics:
- Acoustic Scattering: Zhenyu Tang et al. utilized point clouds to predict how sound waves scatter off dynamic environments. When objects fracture or merge, a point-based network handles the topological disruption seamlessly, bypassing the rigid connectivity constraints that restrict traditional mesh-based acoustic solvers.
- Collision Detection: For rigid body dynamics and robotics, Nico Gürtler et al. showed that representing complex, non-convex shapes as collections of points with associated normals enables vectorized, analytical collision operations. This eliminates the need for manual convex decomposition and maintains a consistent runtime regardless of the geometric complexity.
By treating surfaces as dynamic, unstructured collections of points, modern 3D vision and graphics frameworks achieve a rare trifecta: the explicit spatial localization of meshes, the topological freedom of implicit fields, and a total immunity to the constraints of fixed geometric templates.
Key Foundational Work
A Radial Basis Function (RBF)-Finite Difference (FD) Method for Diffusion and Reaction-Diffusion Equations on Surfaces (2014) — contributed 1 concept elements to this direction.
Using Dirichlet Free Form Deformation to Fit Deformable Models to Noisy 3-D Data (2002) — contributed 1 concept elements to this direction.