Abstract
For more than a decade, deep learning models have been dominating in various 2D imaging tasks. Their application is now extending to 3D imaging, with 3D Convolutional Neural Networks (3D CNNs) being able to process LIDAR, MRI, and CT scans, with significant implications for fields such as autonomous driving and medical imaging. In these critical settings, explaining the model's decisions is fundamental. Despite recent advances in Explainable Artificial Intelligence, however, little effort has been devoted to explaining 3D CNNs, and many works explain these models via inadequate extensions of 2D saliency methods. A fundamental limitation to the development of 3D saliency methods is the lack of a benchmark to quantitatively assess these on 3D data. To address this issue, we propose SE3D: a framework for Saliency method Evaluation in 3D imaging. We propose modifications to ShapeNet, ScanNet, and BraTS datasets, and evaluation metrics to assess saliency methods for 3D CNNs. We evaluate both state-of-the-art saliency methods designed for 3D data and extensions of popular 2D saliency methods to 3D. Our experiments show that 3D saliency methods do not provide explanations of sufficient quality, and that there is margin for future improvements and safer applications of 3D CNNs in critical fields.
Introduction
Deep learning models, particularly 3D Convolutional Neural Networks (3D CNNs), are increasingly used in high-stakes fields like autonomous driving (LiDAR) and medical diagnosis (MRI/CT scans). In these settings, "black box" models are insufficient; understanding the rationale behind a model's decision is critical for safety and trust. Saliency methods, which produce heatmaps highlighting influential input regions, are the primary tools for explainability. However, most current 3D saliency methods are "naive" extensions of 2D techniques, and until now, there has been no rigorous way to measure how well these 3D heatmaps align with ground truth.
SE3D is the first comprehensive benchmarking framework designed to quantitatively evaluate saliency methods in 3D imaging. By introducing specialized 3D metrics and modified datasets (ShapeNet, ScanNet, and BraTS), this framework reveals a significant performance gap: current 3D explanation methods are far less reliable than their 2D counterparts, especially in critical medical applications.
Methodology
The SE3D framework adapts the rigorous evaluation standards of 2D computer vision to the 3D domain. Treating a 3D volume as a stack of 2D slices—the standard approach—fails to capture 3D connectivity, leading to misleading evaluation results.
Key contributions include:
- 3D WSOL Metrics: Extension of Weakly Supervised Object Localization (WSOL) metrics to 3D. For example, Max3DBoxAcc measures the Intersection over Union (IoU) between the 3D bounding box of the saliency map and the ground truth object.
- 3D WSSS Metrics: For voxel-level accuracy, SE3D introduces VxAP (Voxel Average Precision) and MaxF1, measuring how precisely the saliency map covers the target object without bleeding into irrelevant space.
- Mass Concentration (MC): A novel metric designed to detect "Clever Hans" behavior, where a model focuses on irrelevant background noise instead of the actual object of interest.
Benchmarking Datasets
Since standard 3D datasets often lack the specific labels needed for both training (classification) and evaluation (segmentation), the authors created five modified benchmarks to test different aspects of model explainability:
- shapenet-binary & shapenet-pairs: Based on synthetic CAD models. The "pairs" version includes a "noise" object to test if the saliency method can ignore irrelevant data.
- scannet-isolated & scannet-crop: Derived from real-world indoor 3D scans.
- brats-halves: A medical dataset created by splitting brain MRI scans along the symmetry plane to create "tumor" and "no-tumor" samples for binary classification.
Results and Discussion
The quantitative results show a sobering reality for 3D explainability. While 2D saliency methods can achieve a MaxBoxAcc of ~0.78 on standard benchmarks, the best 3D methods often hover between 0.19 and 0.63 on similar tasks.
- The Dimensionality Curse: Localization errors are much more punishing in 3D. An error that might look small in a 2D slice can lead to a massive drop in 3D IoU, explaining why 3D saliency maps often appear less precise.
- Top Performer: Saliency Tubes, a method specifically designed for volumetric data, was the most robust, particularly on the ScanNet datasets (achieving a Max3DBoxAcc of 0.63). In contrast, popular 2D extensions like Grad-CAM tend to produce very dispersed heatmaps.
- Failure in Medical Tasks: On the
brats-halvesbrain tumor dataset, most methods (including Grad-CAM and HiResCAM) achieved a Max3DBoxAcc of 0.00, failing to reliably overlap with the actual tumor location at the 0.5 IoU threshold.
Conclusion
The SE3D framework highlights that current 3D-specific saliency methods do not provide explanations of sufficient quality for critical fields. Future work should focus on developing sophisticated 3D-native methods that can bridge the performance gap, extending the framework to multi-class and spatio-temporal scenarios, and improving reliability in medical contexts.
The code and benchmarks are publicly available at https://github.com/Nexer8/SE3D.
Summary updated