Roughness Index and Roughness Distance for Benchmarking Medical Segmentation

Vidhiwar Singh Rathour, Kashu Yamazaki, T. Hoang Ngan Le

Introduction

In medical imaging, the accurate segmentation of anatomical structures—such as brain tumors or internal organs—is vital for diagnosis and treatment planning. However, standard evaluation metrics like the Dice Score and Hausdorff Distance (HDD) often fail to capture critical surface details. While the Dice Score measures volumetric overlap, it is largely blind to surface texture. Similarly, the HDD measures the maximum distance between two surfaces but remains insensitive to the density of surface irregularities.

This paper introduces a new suite of metrics—Roughness Index (RI) and Roughness Distance (RD)—adapted from civil engineering principles to quantify surface "jaggedness." These metrics aim to improve the evaluation and post-processing of 3D medical images by detecting local surface irregularities like small spikes or holes that traditional metrics overlook.

Methodology: Adapting Surface Profile Logic

The authors look toward civil engineering, where measuring the roughness of materials like concrete is standardized using laser-scanning parameters. They adapt these concepts for the digital voxels of medical scans using the following parameters:

  • Zeta ($\zeta$): Instead of a flat laser plane, the authors use the object's center of gravity as a reference point. $\zeta$ represents the distance from the center to any point on the surface.
  • Roughness Index (RI): This calculates the average deviation of the surface height ($\zeta$) from the mean height within a local window. A high RI indicates a jagged, irregular surface.
  • Roughness Distance (RD/ARD): While RI describes a single object, RD measures the difference in roughness between two objects, such as a ground truth mask and an AI-generated prediction.
Visualization of regular and irregular surface irregularities (spikes and holes) across sequential slices, used to evaluate the sensitivity of the proposed roughness metrics.

The Rough Boolean Matrix and Surface Smoothing

Beyond evaluation, the authors propose a "Rough Boolean Matrix" approach for post-processing. By defining the segmentation space as a union of foreground and background sets, the algorithm identifies specific voxels that are "too rough" compared to their neighbors based on a threshold ($\kappa$). These voxels are then removed or filled, effectively "sanding down" the segmentation to produce a smoother, more anatomically plausible result.

Decomposition of the segmentation space into foreground (F) and background (G) sets, forming the basis for the Rough Boolean Matrix calculation.

Experimental Validation

The metrics were validated using controlled synthetic datasets consisting of 2D images ($100 \times 100$ pixels) and 3D volumes ($100 \times 100 \times 100$ voxels). These models featured spheres and circles modified with "spikes" and "holes" of varying frequencies and lengths. By using synthetic data with known error parameters, the authors could precisely demonstrate the limitations of traditional metrics.

Key Insights and Results

The "Hausdorff Blind Spot"

A primary finding is the visual and quantitative proof that HDD is a "coarse" metric. In experiments, a sphere with a single spike and a sphere with dozens of spikes yielded the exact same HDD score (20.0). In contrast, the proposed Average Roughness Distance (ARD) increased significantly (from 0.0592 to 0.0692), correctly reflecting the increased noise and drop in segmentation quality.

Sensitivity to Error Density

Quantitative results further highlight this sensitivity. When moving from a low-roughness 2D model to a high-roughness model, the Roughness Ratio (RR) jumped from 0.0235 to 0.1377, while the Hausdorff Distance remained unchanged at 20. This demonstrates that RI and RD can distinguish between a mostly smooth surface with a single outlier and a consistently noisy surface.

Effectiveness of Post-Processing

The proposed smoothing algorithm successfully transformed a "star-shaped" noisy segmentation back into a near-perfect circle, reducing the RI from 0.1847 to 0.0417. This suggests that the Roughness Matrix can be used not just for benchmarking, but as an active tool for improving model outputs.

Limitations and Future Work

While promising, the authors note several challenges for clinical adoption:

  1. Alignment Sensitivity: The calculation of $\zeta$ relies on the center of gravity. Significant misalignments between the prediction and ground truth can skew the results.
  2. Hyperparameter Tuning: The choice of "window size" for local roughness is critical. Optimal results were found using 3% to 10% of the image's smallest dimension, but this may vary across different anatomical structures.
  3. Clinical Correlation: Future research is required to determine how these roughness metrics correlate with actual clinical outcomes in complex real-world datasets, such as brain MRIs or lung CT scans.

Summary updated

Abstract

Medical image segmentation is one of the most challenging tasks in medical image analysis and has been widely developed for many clinical applications. Most of the existing metrics have been first designed for natural images and then extended to medical images. While object surface plays an important role in medical segmentation and quantitative analysis i.e. analyze brain tumor surface, measure gray matter volume, most of the existing metrics are limited when it comes to analyzing the object surface, especially to tell about surface smoothness or roughness of a given volumetric object or to analyze the topological errors. In this paper, we first analysis both pros and cons of all existing medical image segmentation metrics, specially on volumetric data. We then propose an appropriate roughness index and roughness distance for medical image segmentation analysis and evaluation. Our proposed method addresses two kinds of segmentation errors, i.e. (i)topological errors on boundary/surface and (ii)irregularities on the boundary/surface. The contribution of this work is four-fold: (i) detect irregular spikes/holes on a surface, (ii) propose roughness index to measure surface roughness of a given object, (iii) propose a roughness distance to measure the distance of two boundaries/surfaces by utilizing the proposed roughness index and (iv) suggest an algorithm which helps to remove the irregular spikes/holes to smooth the surface. Our proposed roughness index and roughness distance are built upon the solid surface roughness parameter which has been successfully developed in the civil engineering.
BibTeX
@misc{rathour2021roughness,
  author = {Vidhiwar Singh Rathour and Kashu Yamazaki and T. Hoang Ngan Le},
  title = {{Roughness Index and Roughness Distance for Benchmarking Medical Segmentation}},
  year = {2021},
  eprint = {2103.12350},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2103.12350},
}
Figure