General Hypernetwork Framework for Creating 3D Point Clouds

Przemyslaw Spurek, Maciej Zieba, Jacek Tabor, Tomasz TrzcinskiIEEE TPAMI 2022 · OpenReview

Abstract

In this work, we propose a novel method for generating 3D point clouds that leverages the properties of hypernetworks. Contrary to existing methods that learn only the representation of a 3D object, our approach simultaneously finds a representation of the object and its 3D surface. The main idea of our HyperCloud method is to build a hypernetwork that returns weights of a particular neural network (target network) trained to map points from a prior distribution into a 3D shape. Consequently, a particular 3D shape can be generated using point-by-point sampling from the prior distribution and transforming the sampled points with the target network. We further extend our method by incorporating flow-based models, resulting in a novel HyperFlow approach.

Introduction

Generating high-quality 3D objects is a cornerstone of modern computer vision, with applications ranging from autonomous driving to robotic manipulation. However, representing 3D data efficiently remains a challenge. Most generative models for 3D data either produce a fixed number of points, limiting flexibility, or rely on complex flow-based architectures that are computationally expensive and memory-intensive. Furthermore, standard point cloud generators often fail to capture the underlying surface of an object, requiring expensive post-processing to produce smooth meshes. This paper introduces a unified framework that addresses these limitations by using hypernetworks to parameterize the 3D surface directly.

Methodology

The core innovation is the use of a hypernetwork to generate the weights of a "target" neural network. Instead of a model that directly outputs 3D coordinates, the hypernetwork outputs the parameters of a function that maps a simple prior distribution (e.g., a 3D ball) into a complex 3D shape.

HyperCloud and HyperFlow

In the HyperCloud architecture, an auto-encoder compresses a 3D shape into a latent vector, which the hypernetwork uses to produce weights for a fully connected target network. HyperFlow extends this by replacing the target network with a Continuous Normalizing Flow (CNF). To stabilize training, the authors introduced a Spherical Log-Normal Distribution, which mimics the topology of 3D objects by concentrating density near a 2D surface.

The Triangulation Trick

Because the target network learns a continuous mapping from a 3D ball to the target shape, the boundary of the ball naturally maps to the object's surface. By feeding the vertices of a standard sphere mesh into the target network, the model "warps" the sphere into the target object's shape, producing a 3D mesh without additional processing.

Reconstructed table mesh demonstrating the continuous surface representation achieved through hypernetwork weights.

Results and Discussion

The models were evaluated on the ShapeNet and ModelNet datasets, focusing on generative tasks and unsupervised classification.

Efficiency and Accuracy

HyperFlow achieved over an order of magnitude reduction in both training time and GPU memory usage compared to PointFlow. Despite this efficiency, the framework produced results comparable to or better than existing methods in terms of Chamfer Distance (CD) and Earth Mover’s Distance (EMD).

High-Quality Mesh Generation

Unlike many competitors that only produce sparse point clouds, the HyperCloud framework generates clean, continuous mesh representations. By predicting weights instead of points, the model effectively learns a unique mathematical function for every object, allowing for "infinite resolution" sampling.

Generated 3D meshes for various object categories (car, table, top, chair) demonstrating the framework's versatility and surface quality.

Conclusion

The HyperCloud and HyperFlow frameworks offer an elegant and flexible approach to 3D shape generation. By treating weights as a parametrization of the surface, the method achieves significant efficiency gains and produces high-quality continuous meshes. Future directions include applying this framework to large-scale LIDAR reconstructions and objects with more complex topologies.

Summary updated

Abstract

In this work, we propose a novel method for generating 3D point clouds that leverages the properties of hypernetworks. Contrary to the existing methods that learn only the representation of a 3D object, our approach simultaneously finds a representation of the object and its 3D surface. The main idea of our HyperCloud method is to build a hypernetwork that returns weights of a particular neural network (target network) trained to map points from prior distribution into a 3D shape. As a consequence, a particular 3D shape can be generated using point-by-point sampling from the prior distribution and transforming the sampled points with the target network. Since the hypernetwork is based on an auto-encoder architecture trained to reconstruct realistic 3D shapes, the target network weights can be considered to be a parametrization of the surface of a 3D shape, and not a standard representation of point cloud usually returned by competitive approaches. We also show that relying on hypernetworks to build 3D point cloud representations offers an elegant and flexible framework. To that point, we further extend our method by incorporating flow-based models, which results in a novel HyperFlow approach.
BibTeX
@article{spurek2022general,
  author = {Przemyslaw Spurek and Maciej Zieba and Jacek Tabor and Tomasz Trzcinski},
  title = {{General Hypernetwork Framework for Creating 3D Point Clouds}},
  journal = {IEEE Trans. Pattern Anal. Mach. Intell.},
  year = {2022},
  url = {https://openreview.net/forum?id=H6JYDVin6jp},
}
Figure