Biomedical image analysis competitions: The state of current participation practice

Matthias Eisenmann, Annika Reinke, Vivienn Weru, Minu Dietlinde Tizabi, Fabian Isensee, Tim J. Adler, Patrick Godau, Veronika Cheplygina, Michal Kozubek, Sharib Ali, Anubha Gupta, Jan Kybic, Alison Noble, Carlos Ortiz de Solórzano, Samiksha Pachade, Caroline Petitjean, Daniel Sage, Donglai Wei, Elizabeth Wilden, Deepak Alapatt, Vincent Andrearczyk, Ujjwal Baid, Spyridon Bakas, Niranjan Balu, Sophia Bano, Vivek Singh Bawa, Jorge Bernal, Sebastian Bodenstedt, Alessandro Casella, Jinwook Choi, Olivier Commowick, Marie Daum, Adrien Depeursinge, Reuben Dorent, Jan Egger, Hannah Eichhorn, Sandy Engelhardt, Melanie Ganz, Gabriel Girard, Lasse Hansen, Mattias Heinrich, Nicholas Heller, Alessa Hering, Arnaud Huaulmé, Hyunjeong Kim, Bennett Landman, Hongwei Bran Li, Jianning Li, Jun Ma, Anne Martel, Carlos Martín-Isla, Bjoern Menze, Chinedu Innocent Nwoye, Valentin Oreiller, Nicolas Padoy, Sarthak Pati, Kelly Payette, Carole Sudre, Kimberlin van Wijnen, Armine Vardazaryan, Tom Vercauteren, Martin Wagner, Chuanbo Wang, Moi Hoon Yap, Zeyun Yu, Chun Yuan, Maximilian Zenk, Aneeq Zia, David Zimmerer, Rina Bao, Chanyeol Choi, Andrew Cohen, Oleh Dzyubachyk, Adrian Galdran, Tianyuan Gan, Tianqi Guo, Pradyumna Gupta, Mahmood Haithami, Edward Ho, Ikbeom Jang, Zhili Li, Zhengbo Luo, Filip Lux, Sokratis Makrogiannis, Dominik Müller, Young-tack Oh, Subeen Pang, Constantin Pape, Gorkem Polat, Charlotte Rosalie Reed, Kanghyun Ryu, Tim Scherr, Vajira Thambawita, Haoyu Wang, Xinliang Wang, Kele Xu, Hung Yeh, Doyeob Yeo, Yixuan Yuan, Yan Zeng, Xin Zhao, Julian Abbing, Jannes Adam, Nagesh Adluru, Niklas Agethen, Salman Ahmed, Yasmina Al Khalil, Mireia Alenyà, Esa Alhoniemi, Chengyang An, Talha Anwar, Tewodros Weldebirhan Arega, Netanell Avisdris, Dogu Baran Aydogan, Yingbin Bai, Maria Baldeon Calisto, Berke Doga Basaran, Marcel Beetz, Cheng Bian, Hao Bian, Kevin Blansit, Louise Bloch, Robert Bohnsack, Sara Bosticardo, Jack Breen, Mikael Brudfors, Raphael Brüngel, Mariano Cabezas, Alberto Cacciola, Zhiwei Chen, Yucong Chen, Daniel Tianming Chen, Minjeong Cho, Min-Kook Choi, Chuantao Xie Chuantao Xie, Dana Cobzas, Julien Cohen-Adad, Jorge Corral Acero, Sujit Kumar Das, Marcela de Oliveira, Hanqiu Deng, Guiming Dong, Lars Doorenbos, Cory Efird, Sergio Escalera, Di Fan, Mehdi Fatan Serj, Alexandre Fenneteau, Lucas Fidon, Patryk Filipiak, René Finzel, Nuno R. Freitas, Christoph M. Friedrich, Mitchell Fulton, Finn Gaida, Francesco Galati, Christoforos Galazis, Chang Hee Gan, Zheyao Gao, Shengbo Gao, Matej Gazda, Beerend Gerats, Neil Getty, Adam Gibicar, Ryan Gifford, Sajan Gohil, Maria Grammatikopoulou, Daniel Grzech, Orhun Güley, Timo Günnemann, Chunxu Guo, Sylvain Guy, Heonjin Ha, Luyi Han, Il Song Han, Ali Hatamizadeh, Tian He, Jimin Heo, Sebastian Hitziger, SeulGi Hong, SeungBum Hong, Rian Huang, Ziyan Huang, Markus Huellebrand, Stephan Huschauer, Mustaffa Hussain, Tomoo Inubushi, Ece Isik Polat, Mojtaba Jafaritadi, SeongHun Jeong, Bailiang Jian, Yuanhong Jiang, Zhifan Jiang, Yueming Jin, Smriti Joshi, Abdolrahim Kadkhodamohammadi, Reda Abdellah Kamraoui, Inha Kang, Junghwa Kang, Davood Karimi, April Khademi, Muhammad Irfan Khan, Suleiman A. Khan, Rishab Khantwal, Kwang-Ju Kim, Timothy Kline, Satoshi Kondo, Elina Kontio, Adrian Krenzer, Artem Kroviakov, Hugo Kuijf, Satyadwyoom Kumar, Francesco La Rosa, Abhi Lad, Doohee Lee, Minho Lee, Chiara Lena, Hao Li, Ling Li, Xingyu Li, Fuyuan Liao, KuanLun Liao, Arlindo Limede Oliveira, Chaonan Lin, Shan Lin, Akis Linardos, Marius George Linguraru, Han Liu, Tao Liu, Di Liu, Yanling Liu, João Lourenço-Silva, Jingpei Lu, Jiangshan Lu, Imanol Luengo, Christina B. Lund, Huan Minh Luu, Yi Lv, Uzay Macar, Leon Maechler, Sina Mansour L., Kenji Marshall, Moona Mazher, Richard McKinley, Alfonso Medela, Felix Meissen, Mingyuan Meng, Dylan Miller, Seyed Hossein Mirjahanmardi, Arnab Mishra, Samir Mitha, Hassan Mohy-ud-Din, Tony Chi Wing Mok, Gowtham Krishnan Murugesan, Enamundram Naga Karthik, Sahil Nalawade, Jakub Nalepa, Mohamed Naser, Ramin Nateghi, Hammad Naveed, Quang-Minh Nguyen, Cuong Nguyen Quoc, Brennan Nichyporuk, Bruno Oliveira, David Owen, Jimut Bahan Pal, Junwen Pan, Wentao Pan, Winnie Pang, Bogyu Park, Vivek Pawar, Kamlesh Pawar, Michael Peven, Lena Philipp, Tomasz Pieciak, Szymon Plotka, Marcel Plutat, Fattaneh Pourakpour, Domen Preložnik, Kumaradevan Punithakumar, Abdul Qayyum, Sandro Queirós, Arman Rahmim, Salar Razavi, Jintao Ren, Mina Rezaei, Jonathan Adam Rico, ZunHyan Rieu, Markus Rink, Johannes Roth, Yusely Ruiz-Gonzalez, Numan Saeed, Anindo Saha, Mostafa Salem, Ricardo Sanchez-Matilla, Kurt Schilling, Wei Shao, Zhiqiang Shen, Ruize Shi, Pengcheng Shi, Daniel Sobotka, Théodore Soulier, Bella Specktor Fadida, Danail Stoyanov, Timothy Sum Hon Mun, Xiaowu Sun, Rong Tao, Franz Thaler, Antoine Théberge, Felix Thielke, Helena Torres, Kareem A. Wahid, Jiacheng Wang, YiFei Wang, Wei Wang, Xiong Wang, Jianhui Wen, Ning Wen, Marek Wodzinski, Ye Wu, Fangfang Xia, Tianqi Xiang, Chen Xiaofei, Lizhan Xu, Tingting Xue, Yuxuan Yang, Lin Yang, Kai Yao, Huifeng Yao, Amirsaeed Yazdani, Michael Yip, Hwanseung Yoo, Fereshteh Yousefirizi, Shunkai Yu, Lei Yu, Jonathan Zamora, Ramy Ashraf Zeineldin, Dewen Zeng, Jianpeng Zhang, Bokai Zhang, Jiapeng Zhang, Fan Zhang, Huahong Zhang, Zhongchen Zhao, Zixuan Zhao, Jiachen Zhao, Can Zhao, Qingshuo Zheng, Yuheng Zhi, Ziqi Zhou, Baosheng Zou, Klaus Maier-Hein, Paul F. Jäger, Annette Kopp-Schneider, Lena Maier-Hein

Abstract

The number of international benchmarking competitions is steadily increasing in various fields of machine learning (ML) research and practice. So far, however, little is known about the common practice as well as bottlenecks faced by the community in tackling the research questions posed. To shed light on the status quo of algorithm development in the specific field of biomedical imaging analysis, we designed an international survey that was issued to all participants of challenges conducted in conjunction with the IEEE ISBI 2021 and MICCAI 2021 conferences (80 competitions in total). The survey covered participants' expertise and working environments, their chosen strategies, as well as algorithm characteristics. A median of 72% challenge participants took part in the survey. According to our results, knowledge exchange was the primary incentive (70%) for participation, while the reception of prize money played only a minor role (16%). While a median of 80 working hours was spent on method development, a large portion of participants stated that they did not have enough time for method development (32%). 25% perceived the infrastructure to be a bottleneck. Overall, 94% of all solutions were deep learning-based. Of these, 84% were based on standard architectures. 43% of the respondents reported that the data samples (e.g., images) were too large to be processed at once. This was most commonly addressed by patch-based training (69%), downsampling (37%), and solving 3D analysis tasks as a series of 2D tasks. K-fold cross-validation on the training set was performed by only 37% of the participants and only 50% of the participants performed ensembling based on multiple identical models (61%) or heterogeneous models (39%). 48% of the respondents applied postprocessing steps.
BibTeX
@misc{eisenmann2022biomedical,
  author = {Matthias Eisenmann and Annika Reinke and Vivienn Weru and Minu Dietlinde Tizabi and Fabian Isensee and Tim J. Adler and Patrick Godau and Veronika Cheplygina and Michal Kozubek and Sharib Ali and Anubha Gupta and Jan Kybic and Alison Noble and Carlos Ortiz de Solórzano and Samiksha Pachade and Caroline Petitjean and Daniel Sage and Donglai Wei and Elizabeth Wilden and Deepak Alapatt and Vincent Andrearczyk and Ujjwal Baid and Spyridon Bakas and Niranjan Balu and Sophia Bano and Vivek Singh Bawa and Jorge Bernal and Sebastian Bodenstedt and Alessandro Casella and Jinwook Choi and Olivier Commowick and Marie Daum and Adrien Depeursinge and Reuben Dorent and Jan Egger and Hannah Eichhorn and Sandy Engelhardt and Melanie Ganz and Gabriel Girard and Lasse Hansen and Mattias Heinrich and Nicholas Heller and Alessa Hering and Arnaud Huaulmé and Hyunjeong Kim and Bennett Landman and Hongwei Bran Li and Jianning Li and Jun Ma and Anne Martel and Carlos Martín-Isla and Bjoern Menze and Chinedu Innocent Nwoye and Valentin Oreiller and Nicolas Padoy and Sarthak Pati and Kelly Payette and Carole Sudre and Kimberlin van Wijnen and Armine Vardazaryan and Tom Vercauteren and Martin Wagner and Chuanbo Wang and Moi Hoon Yap and Zeyun Yu and Chun Yuan and Maximilian Zenk and Aneeq Zia and David Zimmerer and Rina Bao and Chanyeol Choi and Andrew Cohen and Oleh Dzyubachyk and Adrian Galdran and Tianyuan Gan and Tianqi Guo and Pradyumna Gupta and Mahmood Haithami and Edward Ho and Ikbeom Jang and Zhili Li and Zhengbo Luo and Filip Lux and Sokratis Makrogiannis and Dominik Müller and Young-tack Oh and Subeen Pang and Constantin Pape and Gorkem Polat and Charlotte Rosalie Reed and Kanghyun Ryu and Tim Scherr and Vajira Thambawita and Haoyu Wang and Xinliang Wang and Kele Xu and Hung Yeh and Doyeob Yeo and Yixuan Yuan and Yan Zeng and Xin Zhao and Julian Abbing and Jannes Adam and Nagesh Adluru and Niklas Agethen and Salman Ahmed and Yasmina Al Khalil and Mireia Alenyà and Esa Alhoniemi and Chengyang An and Talha Anwar and Tewodros Weldebirhan Arega and Netanell Avisdris and Dogu Baran Aydogan and Yingbin Bai and Maria Baldeon Calisto and Berke Doga Basaran and Marcel Beetz and Cheng Bian and Hao Bian and Kevin Blansit and Louise Bloch and Robert Bohnsack and Sara Bosticardo and Jack Breen and Mikael Brudfors and Raphael Brüngel and Mariano Cabezas and Alberto Cacciola and Zhiwei Chen and Yucong Chen and Daniel Tianming Chen and Minjeong Cho and Min-Kook Choi and Chuantao Xie Chuantao Xie and Dana Cobzas and Julien Cohen-Adad and Jorge Corral Acero and Sujit Kumar Das and Marcela de Oliveira and Hanqiu Deng and Guiming Dong and Lars Doorenbos and Cory Efird and Sergio Escalera and Di Fan and Mehdi Fatan Serj and Alexandre Fenneteau and Lucas Fidon and Patryk Filipiak and René Finzel and Nuno R. Freitas and Christoph M. Friedrich and Mitchell Fulton and Finn Gaida and Francesco Galati and Christoforos Galazis and Chang Hee Gan and Zheyao Gao and Shengbo Gao and Matej Gazda and Beerend Gerats and Neil Getty and Adam Gibicar and Ryan Gifford and Sajan Gohil and Maria Grammatikopoulou and Daniel Grzech and Orhun Güley and Timo Günnemann and Chunxu Guo and Sylvain Guy and Heonjin Ha and Luyi Han and Il Song Han and Ali Hatamizadeh and Tian He and Jimin Heo and Sebastian Hitziger and SeulGi Hong and SeungBum Hong and Rian Huang and Ziyan Huang and Markus Huellebrand and Stephan Huschauer and Mustaffa Hussain and Tomoo Inubushi and Ece Isik Polat and Mojtaba Jafaritadi and SeongHun Jeong and Bailiang Jian and Yuanhong Jiang and Zhifan Jiang and Yueming Jin and Smriti Joshi and Abdolrahim Kadkhodamohammadi and Reda Abdellah Kamraoui and Inha Kang and Junghwa Kang and Davood Karimi and April Khademi and Muhammad Irfan Khan and Suleiman A. Khan and Rishab Khantwal and Kwang-Ju Kim and Timothy Kline and Satoshi Kondo and Elina Kontio and Adrian Krenzer and Artem Kroviakov and Hugo Kuijf and Satyadwyoom Kumar and Francesco La Rosa and Abhi Lad and Doohee Lee and Minho Lee and Chiara Lena and Hao Li and Ling Li and Xingyu Li and Fuyuan Liao and KuanLun Liao and Arlindo Limede Oliveira and Chaonan Lin and Shan Lin and Akis Linardos and Marius George Linguraru and Han Liu and Tao Liu and Di Liu and Yanling Liu and João Lourenço-Silva and Jingpei Lu and Jiangshan Lu and Imanol Luengo and Christina B. Lund and Huan Minh Luu and Yi Lv and Yi Lv and Uzay Macar and Leon Maechler and Sina Mansour L. and Kenji Marshall and Moona Mazher and Richard McKinley and Alfonso Medela and Felix Meissen and Mingyuan Meng and Dylan Miller and Seyed Hossein Mirjahanmardi and Arnab Mishra and Samir Mitha and Hassan Mohy-ud-Din and Tony Chi Wing Mok and Gowtham Krishnan Murugesan and Enamundram Naga Karthik and Sahil Nalawade and Jakub Nalepa and Mohamed Naser and Ramin Nateghi and Hammad Naveed and Quang-Minh Nguyen and Cuong Nguyen Quoc and Brennan Nichyporuk and Bruno Oliveira and David Owen and Jimut Bahan Pal and Junwen Pan and Wentao Pan and Winnie Pang and Bogyu Park and Vivek Pawar and Kamlesh Pawar and Michael Peven and Lena Philipp and Tomasz Pieciak and Szymon Plotka and Marcel Plutat and Fattaneh Pourakpour and Domen Preložnik and Kumaradevan Punithakumar and Abdul Qayyum and Sandro Queirós and Arman Rahmim and Salar Razavi and Jintao Ren and Mina Rezaei and Jonathan Adam Rico and ZunHyan Rieu and Markus Rink and Johannes Roth and Yusely Ruiz-Gonzalez and Numan Saeed and Anindo Saha and Mostafa Salem and Ricardo Sanchez-Matilla and Kurt Schilling and Wei Shao and Zhiqiang Shen and Ruize Shi and Pengcheng Shi and Daniel Sobotka and Théodore Soulier and Bella Specktor Fadida and Danail Stoyanov and Timothy Sum Hon Mun and Xiaowu Sun and Rong Tao and Franz Thaler and Antoine Théberge and Felix Thielke and Helena Torres and Kareem A. Wahid and Jiacheng Wang and YiFei Wang and Wei Wang and Xiong Wang and Jianhui Wen and Ning Wen and Marek Wodzinski and Ye Wu and Fangfang Xia and Tianqi Xiang and Chen Xiaofei and Lizhan Xu and Tingting Xue and Yuxuan Yang and Lin Yang and Kai Yao and Huifeng Yao and Amirsaeed Yazdani and Michael Yip and Hwanseung Yoo and Fereshteh Yousefirizi and Shunkai Yu and Lei Yu and Jonathan Zamora and Ramy Ashraf Zeineldin and Dewen Zeng and Jianpeng Zhang and Bokai Zhang and Jiapeng Zhang and Fan Zhang and Huahong Zhang and Zhongchen Zhao and Zixuan Zhao and Jiachen Zhao and Can Zhao and Qingshuo Zheng and Yuheng Zhi and Ziqi Zhou and Baosheng Zou and Klaus Maier-Hein and Paul F. Jäger and Annette Kopp-Schneider and Lena Maier-Hein},
  title = {{Biomedical image analysis competitions: The state of current participation practice}},
  year = {2022},
  eprint = {2212.08568},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2212.08568},
}

Loading related content...

Figure