Deep learning-enabled 3D multimodal fusion of cone-beam CT and intraoral mesh scans for clinically applicable tooth-bone reconstruction

Jiaxiang Liu, Jin Hao, Hangzheng Lin, Wei Pan, Jianfei Yang, Yang Feng, Gaoang Wang, Jin Li, Zuolin Jin, Zhihe Zhao, Zuozhu Liu

TL;DR

This paper introduces the Deep Dental Multimodal Fusion (DDMF) framework, a novel deep learning approach that automatically fuses cone-beam CT (CBCT) and intraoral scan (IOS) data to create highly accurate 3D tooth-bone reconstructions. This advancement significantly improves virtual dental treatment planning by providing dentists with precise models of crown-root-bone structures, reducing manual effort and time.

What Problem Does This Paper Address?

Accurate 3D reconstruction of teeth and surrounding bone is crucial for modern digital dentistry, enabling precise treatment planning for procedures like orthodontics and implant surgery. Current methods often rely on either CBCT scans, which can suffer from shrinkage and lack detailed occlusal surface information, or intraoral scans (IOS), which primarily capture tooth crowns. Manually fusing these modalities is time-consuming and error-prone. Existing automated fusion methods are either not fully automatic, lack clinical applicability, or haven't been rigorously validated on large-scale, real-world datasets. This paper addresses the need for an efficient, accurate, and automated system to integrate CBCT and IOS data for comprehensive tooth-bone reconstruction.

Methods

The DDMF framework is a multimodal fusion pipeline comprising three main stages:

  • CBCT Segmentation (TSTNet): A novel deep learning model, Tooth Swin Transformer Network (TSTNet), is proposed for segmenting tooth and alveolar bone structures from 2D CBCT slices. TSTNet utilizes a Swin Transformer backbone for feature extraction and incorporates a Tooth Error Calibration (TEC) head to refine boundary segmentation and a novel loss function to handle class imbalance.
  • IOS Segmentation (IOSNet): A deep learning model, IOSNet, is used for segmenting tooth structures from 3D intraoral scan meshes. IOSNet employs Edge-Conv blocks and novel loss functions, including a centroid loss for shape accuracy and a boundary loss for precise tooth-gingiva and tooth-tooth delineation.
  • Multimodal Fusion: This stage integrates the segmented CBCT and IOS data. It involves:
  • Reconstructing 3D CBCT meshes and segmenting them into half-jaws using a novel point curvature feature-based approach.
  • Registering the CBCT and IOS meshes using a two-stage method (global registration with RANSAC and multiscale ICP).
  • Replacing the CBCT tooth crowns with the higher-fidelity IOS crowns to generate the final fused 3D model.

Datasets

The study utilized a large-scale, real-world dataset comprising:

  • 503 CBCT scans and corresponding IOS meshes: Collected from hospitals and clinics in China between 2018 and 2021, featuring patients with malocclusion aged 9-48.
  • 28,559 IOS meshes: Used for evaluating the IOS segmentation module.

The dataset was manually annotated by human experts to ensure clinical applicability.

Key Insights and Ideas

  • Synergistic Fusion: The core idea is that combining the detailed bone structure from CBCT with the precise occlusal surface and crown detail from IOS yields a superior reconstruction than either modality alone.
  • Point Curvature for Segmentation: A novel point curvature feature is introduced to effectively segment half-jaws from CBCT meshes, overcoming challenges posed by connected teeth and bite positions.
  • End-to-End Automation: The DDMF framework automates the entire process from raw scan data to a fused 3D model, significantly reducing manual intervention.
  • Clinical Applicability Focus: The study emphasizes not just algorithmic performance but also the practical utility of the fused models in real-world clinical settings, validated through collaboration with dental professionals.

Key Results

  • CBCT Segmentation (TSTNet): Achieved superior performance with an average Dice coefficient of 93.99% and IoU of 88.68%, outperforming several state-of-the-art baselines.
  • IOS Segmentation (IOSNet): Demonstrated excellent results with mean IoU values of 93.70% and 95.70% for maxillary and mandibular scans, respectively, outperforming baselines by up to 3.52%.
  • Multimodal Fusion: The DDMF framework achieved an average symmetric surface distance (ASSD) of 0.17 mm, a Hausdorff distance (HD) of 0.37 mm, and a Chamfer distance (CD) of 0.20 mm, indicating high accuracy in reconstruction.
  • Efficiency: The DDMF framework generates a fused 3D mesh model in 20-25 minutes, a significant reduction compared to the over 5 hours required by human experts.

Limitations and Future Work

  • Intermediate Step Robustness: The half-jaw separation and registration method, while novel, can still fail in 6-10% of cases, necessitating manual correction. Future work aims to develop more robust learning-based methods for these steps.
  • Segmentation Refinement: While TSTNet and IOSNet show superior performance, further improvements in segmentation accuracy are possible, especially for complex cases with artifacts or unerupted teeth.
  • Clinical Validation: Although integrated into clinical software and validated in real-world cases, further rigorous multicenter clinical trials are suggested to explore clinical usage and dental findings more extensively.
  • Computational Acceleration: The current inference process can be further accelerated by parallel processing of CBCT slices or simultaneous segmentation of IOS and CBCT.

Summary updated

Abstract

High-fidelity three-dimensional (3D) models of tooth-bone structures are valuable for virtual dental treatment planning; however, they require integrating data from cone-beam computed tomography (CBCT) and intraoral scans (IOS) using methods that are either error-prone or time-consuming. Hence, this study presents Deep Dental Multimodal Fusion (DDMF), an automatic multimodal framework that reconstructs 3D tooth-bone structures using CBCT and IOS. Specifically, the DDMF framework comprises CBCT and IOS segmentation modules as well as a multimodal reconstruction module with novel pixel representation learning architectures, prior knowledge-guided losses, and geometry-based 3D fusion techniques. Experiments on real-world large-scale datasets revealed that DDMF achieved superior segmentation performance on CBCT and IOS, achieving a 0.17 mm average symmetric surface distance (ASSD) for 3D fusion with a substantial processing time reduction. Additionally, clinical applicability studies have demonstrated DDMF’s potential for accurately simulating tooth-bone structures throughout the orthodontic treatment process.
BibTeX
@article{liu2023deep,
  author = {Jiaxiang Liu and Jin Hao and Hangzheng Lin and Wei Pan and Jianfei Yang and Yang Feng and Gaoang Wang and Jin Li and Zuolin Jin and Zhihe Zhao and Zuozhu Liu},
  title = {{Deep learning-enabled 3D multimodal fusion of cone-beam CT and intraoral mesh scans for clinically applicable tooth-bone reconstruction}},
  journal = {Patterns},
  year = {2023},
  doi = {10.1016/j.patter.2023.100825},
}
Figure