SRG-Net: Unsupervised Segmentation for Terracotta Warrior Point Cloud with 3D Pointwise CNN methods

Yao Hu, Guohua Geng, Kang Li, Wei Zhou, Xingxing Hao, Xin Cao

TL;DR

This paper introduces SRG-Net, an unsupervised method for segmenting 3D point clouds of Terracotta Warriors. By combining a novel seed-region-growing algorithm with a dynamic graph convolutional neural network and a refinement step, SRG-Net aims to automate the tedious process of fragment identification for restoration, achieving state-of-the-art accuracy with improved efficiency.

What Problem Does This Paper Address?

The restoration of ancient artifacts, particularly the vast number of Terracotta Warrior fragments, is a labor-intensive and time-consuming process for archaeologists. Manually identifying and matching these fragments is challenging due to their sheer quantity and the intricate nature of 3D object reconstruction. The paper addresses the need for an automated solution to segment 3D point cloud data of these warriors into meaningful parts (e.g., head, hand, foot). This would enable the creation of a database to assist archaeologists in matching unearthed fragments with pre-segmented models, significantly improving the efficiency and accuracy of the restoration workflow. While 3D deep learning has seen advancements in classification and reconstruction, unsupervised part segmentation of point clouds remains an underexplored area.

Methods

SRG-Net employs a three-stage approach:

  1. Seed-Region-Growing (SRG) Pre-segmentation: This novel algorithm leverages point cloud coordinates and normal vectors to perform an initial, coarse segmentation. It starts with randomly selected "seeds" and iteratively grows regions by grouping neighboring points that exhibit similar normal directions and are within a certain Euclidean distance. This step aims to capture basic spatial coherence.
  2. SRG-Net (CNN-based Segmentation): This is the core of the method, inspired by DG-CNN (for dynamic graph learning) and FoldingNet (for auto-encoder structure). It uses a dynamic graph convolutional network to learn both local and global features from the point cloud. The network includes an encoder that processes point features and a segmenter that assigns labels. The dynamic graph construction allows the network to adapt its connectivity based on local point neighborhoods.
  3. Refinement: A post-processing step is applied to improve the segmentation results. This involves extracting "superpoints" and then assigning cluster labels based on the majority label of neighboring points, further refining the initial segmentation from the CNN. This step also incorporates a loss function that leverages the coarse segmentation from the SRG algorithm.

Datasets

  • Terracotta Warrior Dataset: A custom dataset of 500 intact terracotta warrior models, each with approximately 2 million points (including XYZ coordinates and vertical normals). For experiments, models were down-sampled to 10,000 points.
  • ShapeNet Dataset: A widely used benchmark dataset for 3D shape analysis, used here to evaluate the robustness and generalizability of SRG-Net.

Key Insights and Ideas

  • Unsupervised Part Segmentation is Crucial for Real-World Applications: The paper highlights that for tasks like artifact restoration, unsupervised methods are highly desirable as obtaining labeled data for every fragment is impractical.
  • Combining Coarse and Fine Segmentation is Effective: The SRG pre-segmentation provides a strong initial grouping, which the subsequent CNN can then refine. This hybrid approach seems more robust than relying solely on a complex deep learning model from scratch.
  • Leveraging Normals and Spatial Proximity: The SRG algorithm's reliance on normal vectors and Euclidean distance is a sensible heuristic for grouping geometrically similar and spatially contiguous parts of an object.
  • Dynamic Graph Networks for Point Clouds: The use of DG-CNN-inspired dynamic graphs allows the network to capture local geometric relationships effectively, which is essential for understanding the structure of complex objects like warriors.
  • Refinement is Key for Accuracy: The paper demonstrates that a post-processing refinement step significantly boosts segmentation accuracy, suggesting that even good CNN outputs can benefit from further geometric reasoning.

Key Results

  • Terracotta Warrior Dataset: SRG-Net achieved state-of-the-art unsupervised segmentation results, outperforming methods like SRG-DGCNN, SRG-PointNet2, and SRG-PointNet in terms of accuracy (mIoU) while consuming significantly less latency. For instance, it improved accuracy by over 5.5% compared to SRG-DGCNN with about half the latency.
  • ShapeNet Dataset: SRG-Net also demonstrated strong performance on the ShapeNet dataset, achieving an overall accuracy (OA) of 78.195%, outperforming other methods, and showing particular strength in segmenting categories like "knife" (5% increase in accuracy).
  • Qualitative Results: Visualizations show that SRG-Net successfully segments the Terracotta Warriors into distinct parts, and also produces good results on various object categories in ShapeNet, indicating its generalizability.

Limitations and Future Work

  • Handling Small Parts: The authors acknowledge that SRG-Net struggles with very small or fragmented parts, such as hands, due to the limited number of points available for learning their characteristics.
  • Future Work: The authors plan to address the limitations with small parts and hope their work will be beneficial for both Terracotta Warrior research and broader point cloud segmentation applications. They also suggest exploring this method for other archaeological artifacts and 3D object segmentation tasks.

Summary updated

Abstract

The repairing work of terracotta warriors in Emperor Qinshihuang Mausoleum Site Museum is handcrafted by experts, and the increasing amounts of unearthed pieces of terracotta warriors make the archaeologists too challenging to conduct the restoration of terracotta warriors efficiently. We hope to segment the 3D point cloud data of the terracotta warriors automatically and store the fragment data in the database to assist the archaeologists in matching the actual fragments with the ones in the database, which could result in higher repairing efficiency of terracotta warriors. Moreover, the existing 3D neural network research is mainly focusing on supervised classification, clustering, unsupervised representation, and reconstruction. There are few pieces of researches concentrating on unsupervised point cloud part segmentation. In this paper, we present SRG-Net for 3D point clouds of terracotta warriors to address these problems. Firstly, we adopt a customized seed-region-growing algorithm to segment the point cloud coarsely. Then we present a supervised segmentation and unsupervised reconstruction networks to learn the characteristics of 3D point clouds. Finally, we combine the SRG algorithm with our improved CNN(convolution neural network) using a refinement method. This pipeline is called SRG-Net, which aims at conducting segmentation tasks on the terracotta warriors. Our proposed SRG-Net is evaluated on the terracotta warrior data and ShapeNet dataset by measuring the accuracy and the latency. The experimental results show that our SRG-Net outperforms the state-of-the-art methods. Our code is available at https://github.com/hyoau/SRG-Net.
BibTeX
@inproceedings{hu2021srg,
  author = {Yao Hu and Guohua Geng and Kang Li and Wei Zhou and Xingxing Hao and Xin Cao},
  title = {{SRG-Net: Unsupervised Segmentation for Terracotta Warrior Point Cloud with 3D Pointwise CNN methods}},
  booktitle = {CSAE},
  year = {2021},
  url = {https://openreview.net/forum?id=FnYl5sZH6x},
}

Loading related content...

Figure