Author Correction: Why rankings of biomedical image analysis competitions should be interpreted with care

Lena Maier-hein, Matthias Eisenmann, Annika Reinke, Sinan Onogur, Marko Stankovic, Patrick Scholz, Tal Arbel, Hrvoje Bogunović, Andrew P. Bradley, Aaron Carass, Carolin Feldmann, Alejandro F. Frangi, Peter M. Full, Bram van Ginneken, Allan Hanbury, Katrin Honauer, Michal Kozubek, Bennett A. Landman, Keno März, Oskar Maier, Klaus Maier-Hein, bjoern menze, Henning Müller, Peter Neher, Wiro J. Niessen, Nasir Rajpoot, G Sharp, Korsuk Sirinukunwattana, Stefanie Speidel, Christian Stock, Danail Stoyanov, Abdel Aziz Taha, Fons van der Sommen, Ching‐Wei Wang, Marc André Weber, Guoyan Zheng, Pierre Jannin, Annette Kopp-SchneiderNature Communications 2019 · DOI · OpenAlex

Abstract In the original version of this Article the values in the rightmost column of Table 1 were inadvertently shifted relative to the other columns. This has now been corrected in the PDF and HTML versions of the Article.

Introduction

The 2018 study "Why rankings of biomedical image analysis competitions should be interpreted with care," published in Nature Communications, fundamentally changed how the medical imaging community views competition "leaderboards." By analyzing the inherent instability and potential biases in these rankings, the authors highlighted the risks of treating competition outcomes as absolute truths. This document provides a formal correction to a data alignment error in the original publication's summary table.

The Fragility of Biomedical Rankings

In biomedical image analysis challenges (such as those held at MICCAI or ISBI), teams compete to produce the most accurate algorithms for tasks like tumor segmentation or organ detection. The original research by Maier-Hein et al. (2018) addressed a critical gap: the "fragility" of these rankings. The study demonstrated that leaderboards are often highly sensitive to "arbitrary" choices made by organizers, including:

  • Metric Selection: The choice between metrics like Dice score and Hausdorff distance can completely reorder a ranking.
  • Aggregation Strategies: The methods used to combine scores across different cases (e.g., mean vs. median) can significantly alter the final standings.
  • Failure Handling: How missing data or algorithm failures are processed can introduce unintended biases.

Details of the Correction: Rectifying Table 1

The primary focus of this correction is the rectification of Table 1 in the original article. In the version published in December 2018, the values in the rightmost column were inadvertently shifted relative to the other columns.

This clerical error meant that the characteristics of the biomedical challenges listed—such as their specific ranking schemes or validation strategies—were incorrectly attributed to the competitions in the table. The correction ensures that the data accurately reflects the authors' comprehensive meta-analysis of challenge rankings.

Data Integrity in Meta-Research

This correction serves as a critical reminder of the importance of data integrity in high-impact publications. For a paper dedicated to the "careful interpretation" of data, ensuring the accuracy of its own summary tables is paramount. In large-scale meta-analyses involving dozens of rows of competition data, a single-cell shift in a spreadsheet can invalidate the comparative context of the entire table. By providing a transparent correction, the authors and Nature Communications ensure that researchers who cite or utilize this data can adjust their own work accordingly.

Conclusion and Future Standards

While the authors acknowledge the error as "inadvertent," the broader limitations discussed in the original work remain highly relevant. Rankings remain a "snapshot" that can change based on metric choice. The authors continue to advocate for more robust and transparent reporting standards in biomedical challenges, such as the BIAS (Biomedical Image Analysis ChallengeS) statement, to prevent the very interpretation errors this paper warns against.

This summary is based on the formal Author Correction published in Nature Communications.

Summary updated

Abstract

In the original version of this Article the values in the rightmost column of Table 1 were inadvertently shifted relative to the other columns. This has now been corrected in the PDF and HTML versions of the Article.
BibTeX
@inproceedings{maierhein2019author,
  author = {Lena Maier-hein and Matthias Eisenmann and Annika Reinke and Sinan Onogur and Marko Stankovic and Patrick Scholz and Tal Arbel and Hrvoje Bogunović and Andrew P. Bradley and Aaron Carass and Carolin Feldmann and Alejandro F. Frangi and Peter M. Full and Bram van Ginneken and Allan Hanbury and Katrin Honauer and Michal Kozubek and Bennett A. Landman and Keno März and Oskar Maier and Klaus Maier-Hein and bjoern menze and Henning Müller and Peter Neher and Wiro J. Niessen and Nasir Rajpoot and G Sharp and Korsuk Sirinukunwattana and Stefanie Speidel and Christian Stock and Danail Stoyanov and Abdel Aziz Taha and Fons van der Sommen and Ching‐Wei Wang and Marc André Weber and Guoyan Zheng and Pierre Jannin and Annette Kopp-Schneider and Bennett Allan Landman and Nasir M. Rajpoot},
  title = {{Author Correction: Why rankings of biomedical image analysis competitions should be interpreted with care}},
  booktitle = {Nature Communications},
  year = {2019},
  doi = {10.1038/s41467-019-08563-w},
}
Figure