PDT: Point Distribution Transformation with Diffusion Models

Jionghao Wang, Cheng Lin, Yuan Liu, Rui Xu, Zhiyang Dou, Xiao-Xiao Long, Hao-Xiang Guo, Taku Komura, Wenping Wang, Xin Li

Abstract

Point-based representations are fundamental to geometric data structures, yet extracting structured, semantically meaningful information from unstructured point clouds remains a significant challenge. We present PDT (Point Distribution Transformation), a novel framework that leverages diffusion models to transform raw geometric distributions into structured outputs. By utilizing a Diffusion Transformer (DiT) architecture coupled with a Point Voxel CNN (PVCNN), PDT effectively correlates source and target distributions through a denoising process. Our method introduces an exponential noise schedule and Sampling Gradient Adjustment (SGA) to ensure high precision in point placement. Extensive experiments demonstrate that PDT successfully generates surface-aligned keypoints, internal skeletal joints, and continuous feature lines, outperforming existing deterministic and generative baselines in both geometric accuracy and semantic relevance.

Introduction

In 3D geometry processing, the ability to extract high-level "blueprints"—such as skeletal joints for animation or feature lines for design—from raw point clouds is essential. Traditionally, this task has been approached using deterministic feed-forward networks that regress point coordinates directly. However, point clouds are inherently unordered and irregular, and the structures they represent are often multimodal; for instance, there may be multiple valid configurations for a character's skeletal joints.

Deterministic models frequently struggle with this ambiguity, resulting in "averaged" outputs that lack local geometric fidelity. PDT addresses this by framing structural extraction as a distribution-to-distribution mapping problem. By treating the target structure as a probability distribution, PDT can capture complex geometric relationships and produce "artist-like" results that are both precise and semantically coherent.

Methodology

Architecture: PVCNN and DiT

PDT utilizes a hybrid architecture designed to capture both local and global geometric context. A Point Voxel CNN (PVCNN) serves as the feature extractor, processing the input "reference" cloud to provide local geometric guidance. These features are then integrated into a Diffusion Transformer (DiT), which manages the global denoising process. Unlike previous generative models that rely on a single global shape vector, PDT maintains an explicit per-point correspondence between the noisy trajectory and the reference surface, ensuring that the transformation remains grounded in the source geometry.

Precision through Exponential Noise Schedules

A critical observation in the development of PDT was that standard linear noise schedules—common in image-based diffusion—are ill-suited for sparse 3D structures. In image generation, minor noise at the end of the diffusion process results in slight blurring; in 3D keypoint generation, it results in "fuzzy" clusters that fail to align with specific vertices. PDT introduces an exponential noise schedule that rapidly reduces noise levels as the process approaches $t \to 0$. This allows points to converge into tight, mathematically precise clusters necessary for downstream mesh operations.

Sampling Gradient Adjustment (SGA)

To further refine the output during inference, PDT employs Sampling Gradient Adjustment (SGA). This technique applies a correction gradient to the denoising steps, nudging points toward specific geometric targets based on the task. For surface-aligned tasks like remeshing, SGA pushes points toward the input mesh surface; for skeletal tasks, it directs points toward the medial axis (the volumetric center) of the shape.

Experimental Results

Remeshing and Keypoint Extraction

Using the Objaverse dataset, PDT was evaluated on its ability to propose vertex locations for remeshing. In a user study involving 24 participants, PDT’s results were preferred over 74% of the time compared to traditional methods like Quadratic Error Metrics (QEM) or Spectral remeshing. Quantitatively, PDT achieved a lower Chamfer Distance (2.82 vs. 3.11 for QEM) while maintaining a more efficient vertex count, demonstrating its ability to capture semantic importance better than purely geometric algorithms.

Skeletal Rigging

On the ModelsResource RigNetv1 dataset, PDT was tasked with predicting internal skeletal joints for articulated characters. PDT significantly outperformed specialized models like RigNet, achieving a joint Intersection over Union (IoU) of 57.5% (compared to RigNet’s 47.0%) and a precision of 84.8% (compared to 70.4%). The generative nature of the model allowed it to handle diverse character topologies—from humans to multi-legged animals—with high consistency.

Feature Line Extraction

Evaluated on the GarmentCode dataset, PDT demonstrated robustness in extracting continuous feature lines from 3D clothing. Notably, the model showed strong generalization capabilities, successfully processing high-density inputs (up to 8,192 points) even when trained on significantly lower-density data. This suggests that the learned distribution transformation is scale-invariant and robust to sampling noise.

Discussion and Limitations

While PDT represents a significant step forward in point distribution transformation, it is computationally intensive. Training the remeshing model required approximately seven days on eight NVIDIA A6000 GPUs. Additionally, the current framework relies on post-process thresholding to cluster generated points into final keypoints. Future work will explore end-to-end connectivity learning to allow the model to simultaneously predict both point locations and their underlying topological graph (e.g., bone connectivity in a skeleton).

Conclusion

PDT offers a powerful new framework for 3D geometry processing by reframing structural extraction as a diffusion-based distribution transformation. By combining the semantic intuition of deep learning with high-precision geometric constraints, PDT enables the automated generation of structured 3D data that meets the standards of professional artistic workflows.

Code and pre-trained models are available at: https://github.com/shanemankiw/PDT

Summary updated

Abstract

Point-based representations have consistently played a vital role in geometric data structures. Most point cloud learning and processing methods typically leverage the unordered and unconstrained nature to represent the underlying geometry of 3D shapes. However, how to extract meaningful structural information from unstructured point cloud distributions and transform them into semantically meaningful point distributions remains an under-explored problem. We present PDT, a novel framework for point distribution transformation with diffusion models. Given a set of input points, PDT learns to transform the point set from its original geometric distribution into a target distribution that is semantically meaningful. Our method utilizes diffusion models with novel architecture and learning strategy, which effectively correlates the source and the target distribution through a denoising process. Through extensive experiments, we show that our method successfully transforms input point clouds into various forms of structured outputs - ranging from surface-aligned keypoints, and inner sparse joints to continuous feature lines. The results showcase our framework's ability to capture both geometric and semantic features, offering a powerful tool for various 3D geometry processing tasks where structured point distributions are desired. Code will be available at this link: https://github.com/shanemankiw/PDT.
BibTeX
@inproceedings{wang2025pdt,
  author = {Jionghao Wang and Cheng Lin and Yuan Liu and Rui Xu and Zhiyang Dou and Xiao-Xiao Long and Hao-Xiang Guo and Taku Komura and Wenping Wang and Xin Li},
  title = {{PDT: Point Distribution Transformation with Diffusion Models}},
  booktitle = {SIGGRAPH (Conference Paper Track)},
  year = {2025},
  doi = {10.1145/3721238.3730717},
}
Figure