FrameSeg3D: Learned Local Frames for Segmenting Small Parts on High-Resolution 3D Scans

Xinwen Zhang, Tongyan Wang, Huawei LinarXiv 2026 · arXiv

TL;DR

FrameSeg3D is a two-stage framework that solves the challenge of segmenting tiny parts on massive 3D scans by first predicting a local coordinate frame for each part. By normalizing every part to its own high-resolution "canonical view," a single network can achieve state-of-the-art accuracy across diverse domains like fingernails and teeth, even with very limited training data.

What Problem Does This Paper Address?

High-resolution 3D scans, such as those from intraoral dental scanners or hand scanners, often contain millions of vertices. However, the parts we actually care about—like a single tooth or a fingernail—might occupy less than 0.3% of the total scan.

This creates a "needle in a haystack" problem for traditional deep learning:

  1. Memory Constraints: Point-based or mesh-based networks can't fit millions of vertices into GPU memory, forcing them to downsample the scan. This erases the subtle geometric details (like tiny changes in height or curvature) that define part boundaries.
  2. Viewing Geometry: Multi-view methods that render the whole object often see small parts at "grazing angles" (from the side). For example, on a flat hand scan, the thumb nail is tilted significantly compared to the other fingers, leading to distorted views and poor segmentation.
  3. Domain Specificity: Existing two-stage methods often rely on domain-specific priors (like the shape of a dental arch) that don't generalize to other types of parts.

Methods

FrameSeg3D introduces a "predict-then-segment" pipeline that treats every part as if it were the center of its own universe.

  • Stage 1: Frame Prediction: The system first looks at a low-resolution, top-down "overview" of the entire scan. A U-Net-based network predicts a heatmap for part centers, a normal map for their orientations, and a name map. Together, these define a local frame (center and facing direction) for every detected part.
  • Stage 2: In-frame Segmentation: For each predicted frame, the system renders 12 high-resolution 2D views (at a crisp 0.08 mm/pixel) from cameras arranged on a Fibonacci spiral around the part's predicted normal. A 2D segmentation network processes these views, and the resulting masks are "lifted" back onto the 3D mesh vertices.
  • Refinement: Because the segmentation network is trained on "perturbed" frames (intentionally shifted and tilted during training), it is robust to slight errors from the first stage. A final re-centering step closes the loop, using the initial segmentation to refine the frame for a second, more accurate pass.
Overview of the FrameSeg3D pipeline showing frame prediction (a) and in-frame segmentation (b).

Datasets

The authors evaluated FrameSeg3D on two distinct domains:

  • Nails: A newly utilized dataset of 95 high-resolution hand scans (approx. 2 million vertices each) with manual annotations for fingernails.
  • Teeth: The Teeth3DS+ intraoral scan dataset and the 3DTeethSeg’22 challenge private test set, which includes complex cases like crowded or missing teeth.

Key Insights and Ideas

  • Viewing Geometry > Point Density: A core finding is that accuracy is limited more by how a part is seen (direction and scale) than by how many raw points land on it. Centering a view on a part's own normal significantly boosts performance, especially for tilted parts like thumbs.
  • The Power of Normalization: By transforming every part into a fixed scale and orientation, the segmentation task becomes "canonical." This allows a single network to learn features that apply to both nails and teeth simultaneously, effectively sharing knowledge across domains.
  • Finer than the Mesh: By rendering at 0.08 mm/pixel, the system actually sees details finer than the original vertex spacing of the scan, allowing it to trace boundaries with sub-millimeter precision.

Key Results

  • State-of-the-Art Performance: FrameSeg3D achieved a mean Dice score of 0.970 on hand scans and a per-tooth IoU of 96.5 on the Teeth3DS+ test fold, outperforming specialized dental SOTA methods.
  • Precise Boundaries: It excelled at drawing accurate boundaries, reaching a Boundary IoU of 70.5 on teeth—a significant jump over previous methods that often "blur" the lines between teeth and gums.
  • Extreme Data Efficiency: On the nail dataset, a FrameSeg3D model trained on only 8 people outperformed a whole-hand baseline trained on the full set of 30 people.
Qualitative comparison of FrameSeg3D against baselines on hand and jaw scans.

Limitations and Future Work

  • The Frame Network is the Weak Link: While the segmentation stage is highly efficient, the frame prediction stage requires more labeled data and can occasionally miss rare parts, such as third molars (wisdom teeth), if they weren't well-represented in the training set.
  • Inference Speed: The multi-view rendering process, while accurate, is computationally intensive. It takes about 15.5 seconds per hand and 21.9 seconds per jaw, which is slower than some specialized, lower-resolution dental methods.
  • Future Directions: The authors suggest moving toward a fully differentiable pipeline. This would allow the system to learn to predict frames directly from the segmentation loss, potentially removing the need for explicit frame labels and improving performance on rare or unusual parts.

Summary updated

Abstract

Segmenting small parts on high-resolution 3D scans, such as nails on hand scans and teeth on intraoral scans, poses three challenges: 1) each part covers a tiny fraction of the scan, 2) its boundary is only a subtle change of height and curvature, and 3) it faces its own direction. Existing methods fall into two groups, and both fall short. The first group processes the whole scan at once and sees each part too coarsely. Point and mesh networks must shrink the scan to fit in memory, so a small part keeps only a few of its points. Multi-view methods render images of the whole object, so each part fills only a few pixels and a tilted part is seen from the side. The second group first finds each part and then segments it up close. But these methods find the parts with knowledge about one kind of part, for example the shape of the dental arch for teeth. Such knowledge does not carry over to other kinds of parts. We find that accuracy is limited by how a part is seen, from which direction and at what scale, not by how many points land on it. We therefore present FrameSeg3D. It first predicts a local frame for every part, its center and the direction it faces, from a low-resolution view of the whole scan. It then segments the part in views around its frame normal at the scanner's resolution, so covering the whole scan no longer costs resolution. Because every part arrives centered, facing the camera, and at a fixed scale, one segmentation network can serve different kinds of parts and learn from few labeled masks. We test FrameSeg3D on two kinds of parts, nails and teeth. It outperforms state-of-the-art methods on hand scans (mean Dice 0.970 on nails) and on the Teeth3DS+ test fold (per-tooth IoU 96.5), with the most accurate tooth boundaries (boundary IoU 70.5).
BibTeX
@misc{zhang2026frameseg,
  author = {Xinwen Zhang and Tongyan Wang and Huawei Lin},
  title = {{FrameSeg3D: Learned Local Frames for Segmenting Small Parts on High-Resolution 3D Scans}},
  year = {2026},
  eprint = {2611.22124},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.22124},
}
Figure