Observer performance in semi-automated microbleed detection

Hugo J. Kuijf, Manon Brundel, Jeroen de Bresser, Max A. Viergever, Geert Jan Biessels, Mirjam I. Geerlings, Koen L. VinckenProceedings of SPIE, the International Society for Optical Engineering/Proceedings of SPIE 2013 · DOI

TL;DR

This paper introduces a semi-automated method, the Radial Symmetry Transform (RST), to improve the detection of cerebral microbleeds on high-resolution 7.0 T MRI scans. The RST significantly boosts inter-observer agreement and drastically reduces the time needed for detection, making it a promising tool for clinical research and diagnosis.

What Problem Does This Paper Address?

Cerebral microbleeds (CMBs) are small lesions in the brain linked to vascular disease and dementia. Detecting them accurately with MRI is crucial, but the current standard—manual visual rating by experts—is a laborious, time-consuming, and often inconsistent process. This is particularly challenging with high-resolution 7.0 T MRI scans, which offer greater detail but also present more subtle lesions and potential mimics, leading to variability between observers. The paper aims to address this by evaluating a semi-automated approach to streamline and standardize CMB detection.

Methods

The core of the study is the evaluation of the Radial Symmetry Transform (RST), a technique originally designed for highlighting circular objects in 2D images, adapted here for 3D MRI data. The RST works by analyzing gradient information within the MRI data to identify potential microbleed locations. Specifically, it counts neighboring voxels where the gradient points towards the current voxel, assigning a high score to the center of symmetrical, circular structures like microbleeds.

The process involves:

  1. 3D Gradient Computation: Calculating gradient information across the 3D MRI volume.
  2. Symmetry Counting: For each voxel, determining how many neighboring voxels exhibit gradient directions pointing towards it within a defined radius.
  3. Echo Combination: Integrating results from both the first and second echoes of a dual-echo MRI sequence to enhance detection.
  4. Thresholding: Applying thresholds (θ₁ and θ₂) to the RST output to identify candidate microbleed locations.

The study then compares the performance of human observers with and without the RST support. Two experienced neuroradiologists (observers 1 and 2) performed initial visual ratings. Subsequently, two different observers (observers 3 and 4), who were not involved in the initial rating, used a dedicated tool to review and censor the candidate locations identified by the RST.

Datasets

The study utilized MRI data from 18 participants who had previously been identified as having microbleeds from the Second Manifestations of ARTerial disease (SMART) study. These participants were scanned using a 7.0 T dual-echo gradient echo MRI sequence. The data served as the basis for both the initial visual rating and the subsequent evaluation of the RST-assisted detection. The paper does not introduce a new benchmark dataset but rather uses existing patient data to evaluate the RST's performance against a "silver standard" ground truth derived from consensus visual ratings.

Key Insights and Ideas

  • RST as a "Spotlight": The RST acts as a powerful tool to direct observers' attention to potential microbleed locations, significantly reducing the search space and the cognitive load associated with manual scanning.
  • Bridging the Gap: The RST effectively bridges the gap between fully automated and purely manual detection, offering a balance of efficiency and expert oversight.
  • Improving Consistency: By guiding observers to the same potential lesions, the RST dramatically improves inter-observer agreement, a critical factor for reliable clinical interpretation and research.
  • Time is of the Essence: The substantial reduction in rating time per participant is a major practical advantage, enabling larger studies and more efficient clinical workflows.

Key Results

  • Improved Sensitivity: The average sensitivity of observers increased from 59% (without RST) to 69% (with RST).
  • Dramatically Increased Inter-Observer Agreement:
    • Intra-class Correlation Coefficient (ICC) rose from 0.85 to 0.98.
  • Dice's coefficient improved from 0.64 to 0.96.
  • Significant Time Reduction: The time required for rating decreased from an average of 30 minutes per participant (visual rating) to approximately 2 minutes per participant (RST-assisted).
  • Enhanced Ground Truth: The RST helped identify 12 additional microbleeds that were missed in the initial visual rating, leading to a revised ground truth of 66 microbleeds.
  • Fine-tuning Potential: By adjusting RST parameters, sensitivities up to 90% can be achieved, though this comes at the cost of more false positives.

Limitations and Future Work

  • Ground Truth Bias: The authors acknowledge that the inclusion of microbleeds detected by the RST into the ground truth might introduce a bias when evaluating the RST's sensitivity. They refer to this revised ground truth as a "silver standard" rather than a gold standard due to the inherent difficulties in visual rating.
  • False Positives: While the RST improves sensitivity, it also generates a significant number of false positives. Fine-tuning parameters can increase sensitivity but also increases the workload for censoring these false positives.
  • Subjectivity Remains: Despite improvements, some degree of subjectivity in identifying true microbleeds versus mimics still exists, meaning a perfect inter-observer agreement of 1.0 is unlikely. The final decision remains with the medical expert.
  • Future Directions:
    • Reducing the number of false positives generated by the RST.
  • Enabling individual observers to achieve higher sensitivities without relying on multiple consensus meetings.
  • Potentially making censoring by multiple observers superfluous as inter-observer agreement increases.

Summary updated

Abstract

Cerebral microbleeds are small bleedings in the human brain, detectable with MRI. Microbleeds are associated with vascular disease and dementia. The number of studies involving microbleed detection is increasing rapidly. Visual rating is the current standard for detection, but is a time-consuming process, especially at high-resolution 7.0 T MR images, has limited reproducibility and is highly observer dependent. Recently, multiple techniques have been published for the semi-automated detection of microbleeds, attempting to overcome these problems. In the present study, a 7.0 T dual-echo gradient echo MR image was acquired in 18 participants with microbleeds from the SMART study. Two experienced observers identified 54 microbleeds in these participants, using a validated visual rating scale. The radial symmetry transform (RST) can be used for semi-automated detection of microbleeds in 7.0 T MR images. In the present study, the results of the RST were assessed by two observers and 47 microbleeds were identified: 35 true positives and 12 extra positives (microbleeds that were missed during visual rating). Hence, after scoring a total number of 66 microbleeds could be identified in the 18 participants. The use of the RST increased the average sensitivity of observers from 59% to 69%. More importantly, inter-observer agreement (ICC and Dice's coefficient) increased from 0.85 and 0.64 to 0.98 and 0.96, respectively. Furthermore, the required rating time was reduced from 30 to 2 minutes per participant. By fine-tuning the RST, sensitivities up to 90% can be achieved, at the cost of extra false positives.
BibTeX
@inproceedings{kuijf2013observer,
  author = {Hugo J. Kuijf and Manon Brundel and Jeroen de Bresser and Max A. Viergever and Geert Jan Biessels and Mirjam I. Geerlings and Koen L. Vincken},
  title = {{Observer performance in semi-automated microbleed detection}},
  booktitle = {Proceedings of SPIE, the International Society for Optical Engineering/Proceedings of SPIE},
  year = {2013},
  doi = {10.1117/12.2001798},
}
Figure